Fluentd v1.0

  • Logging is a critical component on production environments that allow us to perform monitoring and data analysis. While applications runs at scale, the Logging layer also needs to scale and year over year we see new challenges that needs to be solved from different angles such as parsing, performance, log enrichment (metadata), filtering and so on.

    Fluentd was born to solve Logging problems as a whole, not only for standalone applications but also for distributed architectures where each running application and system have their own way to solve logging, integration between all components and the ability to move data from one place to another in a secure and reliable way was a requirement from day one… and it continue to be as of today. That’s why it has been adopted by thousands of companies and thanks to it growing community around the world demonstrate that Fluentd became the Industry Standard for Logging.

    Fast iterations and solving the problem better than yesterday is our mantra, that’s why the global team of maintainers is proud to announce that Fluentd v1.0 has been released!, this is a big milestone for everyone around the Fluentd Ecosystem which means much more that a number, it’s a maturity resulting of years of work in the community where feedback, ideas and general testing have been the roots for it growth. So thank all you who have been involved on this process!

    What’s Hot in Fluentd v1.0 ?

    Fluentd v1.0 built on top of v0.14-stable, some of the biggest changes on this new series are:

    • Core:
      • Multiprocess Workers: take advantage of SMP systems
      • Sub-second time resolution: all log records now have a granular time resolution
    • Portability: Windows Support has finally arrived to Fluentd.
    • New plugins API: Our biggest contributions to Fluentd ecosystem is through plugins, with more than 700 plugins available made by the community we have focused into improve the developer experience.
    • Data Management: new internal buffers can optionally enable compressions to save disk space.
    • Networking:
      • New Fluentd Forward Protocol v1: includes authentication using shared keys and authorizations through username/password.
      • Native Transport-Layer-Security (TLS) support

    These changes are not the only ones, there many improvements around performance, portability and flexibility for data management.

    Monitoring and Data Streaming

    Fluentd is more than a project, it’s a full ecosystem and integration with third party components is fundamental, that’s why as part of our Fluentd v1.0 release we are proud to announce also the continuous investment in integration in Prometheus (monitoring) and Apache Kafka (data streaming) within many others.

    Prometheus makes easier monitoring and having Fluentd provide native support for it has been in high demand in the last time and we are happy to announce that fluent-plugin-prometheus is now officially part of Fluentd Ecosystem hosted on CNCF Fluent organization on Github.

    Data streaming is such important as monitoring, and companies around are looking for ways to integrate more components where Logging can be a critical compoment of the data pipeline. Fluentd is getting better and better at it: Fluentd and Apache Kafka can talk each other smoothly and securely.

    What’s Next

    Next year we will continue working towards performance improvements and connectors that makes easier to hook Fluentd on any place. As well from an ecosystem perspective, Fluent Bit, our lightweight log processor will keep growing in terms of capabilities for cloud native environments such as like load balancing, persistent queues and monitoring within others.

    2018 will be an exciting journey, don’t hesitate to be part of it!

    Connect at CloudNativeCon + KubeCon

    If you are attending KubeCon + CloudNativeCon North America this week, please join us December 6th for a Fluentd Salon, our community will be presenting the following lightning talks:

    • “Moving Data at Scale” by Sadayuki Furuhashi (Fluentd Creator)
    • “Fluentd and Kafka” by Masahiro Nakagawa (Fluentd Maintainer)
    • “Fluentd and Prometheus” by Yuki Ito (Fluentd Maintainer)
    • “Fluent Bit” by Eduardo Silva (Fluent Bit Maintainer)
    • “Plugging into fluent-bit: how to use plugin templates to customize fluent-bit to serve your needs” by Yeni Capote, Samsung SDS

    we will also discuss Fluentd v1.0, roadmap and tools around the Fluent ecosystem plus an open space for discussions. Additionally, do not miss the additional sessions on Fluentd:

    Closing Thoughts

    Reaching v1.0 for us it’s a new begginning, we will continue working together with our end-users, community and companies around to make Fluentd better for 2018. If you are around CloudNativeCon+KubeCon don’t hesitate to contact us in our sessions or during the break ours, come and join us!, you can also help to solve the problem better than yesterday !

    Let’s keep improving the Logging Standard…

    The post Fluentd v1.0 appeared first on The Linux Foundation.


  • Make ISO from DVD

    In this case I had an OS install disk which was required to be on a virtual node with no optical drive, so I needed to transfer an image to the server to create a VM

    Find out which device the DVD is:



    NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 464.8G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 11.8G 0 lvm [SWAP] └─centos-home 253:2 0 403G 0 lvm /home sdb 8:16 1 14.5G 0 disk /mnt sr0 11:0 1 4.1G 0 rom /run/media/rick/CCSA_X64FRE_EN-US_DV5

    Therefore /dev/sr0 is the location , or disk to be made into an ISO

    I prefer simplicity, and sometimes deal with the fallout after the fact, however Ive repeated this countless times with success.

    dd if=/dev/sr0 of=win10.iso

    Where if=Input file and of=output file

    I chill out and do something else while the image is being copied/created, and the final output:

    8555456+0 records in 8555456+0 records out 4380393472 bytes (4.4 GB) copied, 331.937 s, 13.2 MB/s


    read more
  • Recreate postrgresql database template encode to ASCII

    UPDATE pg_database SET datistemplate = FALSE WHERE datname = 'template1';

    Now we can drop it:

    DROP DATABASE template1;

    Create database from template0, with a new default encoding:

    CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UNICODE'; UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template1'; \c template1 VACUUM FREEZE;

    read more