NOAA’s Mission Toward Open Data Sharing



  • Open Data

    NOAA is working to make all of its data available to an even wider group of people and make it more easily understood (Image: NOAA).

    The goal of the National Oceanic and Atmospheric Administration (NOAA) is to put all of its data — data about weather, climate, ocean coasts, fisheries, and ecosystems – into the hands of the people who need it most. The trick is translating the hard data and making it useful to people who aren’t necessarily subject matter experts, said Edward Kearns, the NOAA’s first ever data officer, speaking at the recent Open Source Leadership Summit (OSLS).

    NOAA’s mission is similar to NASA’s in that it is science based, but “our mission is operations; to get the quality information to the American people that they need to run their businesses, to protect their lives and property, to manage their water resources, to manage their ocean resources,” said Kearns, during his talk titled “Realizing the Full Potential of NOAA’s Open Data.”

    He said that NOAA was doing Big Data long before the term was coined and that the agency has way too much of it – to the tune of 30 petabytes in its archives with another 200 petabytes of data in a working data store. Not surprisingly, NOAA officials have a hard time moving it around and managing it, Kearns said.

    Data Sharing

    NOAA is a big consumer of open source and sharing everything openly is part of the organization’s modus operandi. On a global level, “the agency has been a leader for the entire United States in trying to broker data sharing among countries,” Kearns said. One of the most successful examples has been through the United Nations, with an organization called World Meteorological Organization (WMO).

    Agency officials have a tendency to default making their products accessible in the public domain, something Kearns said he’d like to change. By adopting some modern licensing practices, he believes the NOAA could actually share even more information with the public. “The Linux Foundation has made progress on the community data license agreement. This is one the things I’d like to possibly consider adopting for our organization,’’ he added.

    One of the great success stories the NOAA has in terms of getting critical data to the public was after Hurricane Irma hit Florida in September 2017, he said.

    “As you can imagine, there were a lot of American citizens that were hungry for information and were hitting the NOAA websites very hard and data sites very hard,’’ he said. “Typically, we have a hard time keeping up with that kind of demand.” The National Hurricane Center is part of the NOAA, and the agency took the NHC’s website and put it on Amazon Cloud.

    This gave the agency the ability to handle over a billion hits a day during the peak hurricane season. But, he continued, “we are still … just starting to get into how to adopt some of these more modern technologies to do our job better.”

    Equal Access

    Now the NOAA is looking to find a way to make the data available to an even wider group of people and make it more easily understood. Those are their two biggest challenges: how to disseminate data and how to help people understand it, Kearns said.

    “We’re getting hammered every day by a lot of companies that want the data… and we have to make sure everybody’s got an equal chance of getting the data,” he said.

    This is becoming a harder job because demand is growing exponentially, he said. “Our costs are going up because we need more servers, we need more networks,” and it’s a problem due to budget constraints.

    The agency decided that partnering with industry would help facilitate the delivery of data.

    The NOAA is going into the fourth year of a deal it signed with Amazon, Microsoft, IBM, Google, and a nonprofit out of the University of the Chicago called the Open Commons Consortium (OCC), Kearns said. The agreement is that NOAA data will remain free and open and the OCC will host it at no cost to taxpayers and monetize services around the data.

    The agency is using an academic partner acting as a data broker to help it “flip this data and figure out how to drop it into all of our collaborators’ cloud platforms, and they turn it around and serve many consumers from that,” Kearns explained. “We went from a one-to-many, to a one-to-a-few, to a many model of distribution.”

    People trust NOAA’s data today because they get it from a NOAA data service, he said. Now the agency is asking them to trust the NOAA data that exists outside the federal system on a partner system.

    On AWS alone the NOAA has seen an improvement of over two times the number of people who are using the data, he said. The agency in turn, has seen a 50 percent reduction in hits on the NOAA servers.

    Google has loaded a lot of the agency’s climate data to its BigQuery data warehouse, “and they’ve been able to move petabytes of this data just in a few months, just because the data now has been loaded into a tool people are already using.”

    This “reduces that obstacle of understanding,’’ Kearns noted. “You don’t have to understand a scientific data format, you can go right into BigQuery… and do analyses.”

    Data Trust

    Being able to trust data is also an important component of any shared initiative, and through the NOAA’s Big Data Project, the agency is seeking ways of ensuring that the trust that comes with the NOAA brand is conveyed with the data, he said, so people continue to trust it as they use it.

    “We have a very proud history of this open data leadership, we’re continuing on that path, and we’re trying to see how we can amplify that,’’ Kearns said.

    NOAA officials are now wondering if the data is being made available through these modern cloud platforms will make it easier for users to create information products for themselves and their customers.

    “Of course, we’re also looking for other ways of just doing our business better,’’ he added. But they want to figure out if it makes sense to continue this experiment with its partners. That, he said, they will likely know by early next year.

    Watch the complete presentation below:

    The post NOAA’s Mission Toward Open Data Sharing appeared first on The Linux Foundation.

    https://www.linuxfoundation.org/blog/noaas-mission-toward-open-data-sharing/





Tmux Commands

screen and tmux

A comparison of the features (or more-so just a table of notes for accessing some of those features) for GNU screen and BSD-licensed tmux.

The formatting here is simple enough to understand (I would hope). ^ means ctrl+, so ^x is ctrl+x. M- means meta (generally left-alt or escape)+, so M-x is left-alt+x

It should be noted that this is no where near a full feature-set of either group. This - being a cheat-sheet - is just to point out the most very basic features to get you on the road.

Trust the developers and manpage writers more than me. This document is originally from 2009 when tmux was still new - since then both of these programs have had many updates and features added (not all of which have been dutifully noted here).

Action tmux screen
start a new session tmux OR
tmux new OR
tmux new-session
screen
re-attach a detached session tmux attach OR
tmux attach-session
screen-r
re-attach an attached session (detaching it from elsewhere) tmux attach -d OR
tmux attach-session -d
screen -dr
re-attach an attached session (keeping it attached elsewhere) tmux attach OR
tmux attach-session
screen -x
detach from currently attached session ^b d OR
^b :detach
^a ^d OR
^a :detach
rename-window to newname ^b , <newname> OR
^b :rename-window <newn>
^a A <newname>
list windows ^b w ^a w
list windows in chooseable menu ^a "
go to window # ^b # ^a #
go to last-active window ^b l ^a ^a
go to next window ^b n ^a n
go to previous window ^b p ^a p
see keybindings ^b ? ^a ?
list sessions ^b s OR
tmux ls OR
tmux list-sessions
screen -ls
toggle visual bell ^a ^g
create another window ^b c ^a c
exit current shell/window ^d ^d
split window/pane horizontally ^b " ^a S
split window/pane vertically ^b % ^a |
switch to other pane ^b o ^a <tab>
kill the current pane ^b x OR (logout/^D)
collapse the current pane/split (but leave processes running) ^a X
cycle location of panes ^b ^o
swap current pane with previous ^b {
swap current pane with next ^b }
show time ^b t
show numeric values of panes ^b q
toggle zoom-state of current pane (maximize/return current pane) ^b z
break the current pane out of its window (to form new window) ^b !
re-arrange current panels within same window (different layouts) ^b [space]
Kill the current window (and all panes within) ^b killw [target-window]
  • Open Source Summit

    Join us in Edinburgh! Submit a proposal to speak by July 1 for Open Source Summit & ELC + OpenIoT Summit Europe.

    Submit a proposal to speak at Open Source Summit Europe & ELC + OpenIoT Summit Europe, taking place October 22-24, 2018, in Edinburgh, UK, and share your knowledge and expertise with 2,000+ open source technologists and community leaders. Proposals are being accepted through 11:59pm PDT, Sunday, July 1.

    This year’s tracks and content will cover the following areas at Open Source Summit Europe:

    Cloud Native Apps/Serverless/Microservices Infrastructure & Automation (Cloud/Cloud Native/DevOps) Linux Systems Artificial Intelligence & Data Analytics Emerging Technologies & Wildcard (Networking, Edge, IoT, Hardware, Blockchain) Community, Compliance, Governance, Culture, Open Source Program Management (Open Collaboration Conference track) Diversity & Inclusion (Diversity Empowerment Summit) Innovation at Apache/Apache Projects TODO / Open Source Program Management

    View the full list of suggested topics for Open Source Summit Europe.

    Suggested Embedded Linux Conference (ELC) Topics:

    Audio, Video, Streaming Media and Graphics Security System Size, Boot Speed Real-Time Linux – Performance, Tuning and Mainlining SDKs for Embedded Products Flash Memory Devices and Filesystems Build Systems, Embedded Distributions and Development Tools Linux in Devices such as Mobile Phones, DVRs, TVs, Cameras, etc Use of Linux in Automotive Drones and Robots Linux in the Internet of Things Practical Experiences and War Stories Standards Public Infrastructure Industrial Automation

    This year’s tracks and content will cover the following areas at ELC:

    Suggested OpenIoT Summit Topics:

    Real-Time OS (Zephyr, RIOT, MyNewt, FreeRTOS, NuttX, mbed and Others) Outside World Meets IoT (Sensor Interaction, Low Footprint, Connected Sensors, EMF/RFI Impact) Bootloaders, Firmware & Updates Containers Distributed Edge Application Technologies On-device Analytics Blockchain for Constrained Devices Device Management Power Management Configuration Management Developing for Security Safety Considerations Certifications – Lessons Learned Taking Devices to Product

    View the full list of suggested topics for ELC + OpenIoT Summit Europe.

    SUBMIT FOR OPEN SOURCE SUMMIT EUROPE »SUBMIT FOR ELC + OPENIOT SUMMIT EUROPE »

    Sign up to receive updates on Open Source Summit Europe and ELC + OpenIoT Summit Europe:

    Register & Save

    Not submitting, but plan to attend? Register before August 18 and save $300 with early bird pricing. One registration gets you access to both Open Source Summit Europe & ELC + OpenIoT Summit Europe.

    Interested in Sponsoring?

    Showcase your thought leadership among a vibrant open source community and connect with top influencers driving today’s technology purchasing decisions. Learn how to become a sponsor of Open Source Summit Europe or ELC + OpenIoT Summit Europe.

    The post Last Chance to Speak at Open Source Summit and ELC + OpenIoT Summit Europe – Submit by July 1 appeared first on The Linux Foundation.

    https://www.linuxfoundation.org/blog/last-chance-to-speak-at-open-source-summit-and-elc-openiot-summit-europe-submit-by-july-1/

    read more
  • Open Source Guides

    The Open Source Guides for the Enterprise are now available in Chinese.

    The popular Open Source Guides for the Enterprise, developed by The Linux Foundation in collaboration with the TODO Group, are now available in Chinese. This set of guides provides industry-proven best practices to help organizations successfully leverage open source.

    “Making these resources available to Chinese audiences in their native language will encourage even greater adoption of and participation with open source projects,” said Chris Aniszczyk, CTO of Cloud Native Computing Foundation and co-founder of the TODO Group. The guides span various stages of the open source project lifecycle, from initial planning and formation to winding down a project.

    The 10 guides now available in Mandarin include topics such as:

    Creating an Open Source Program by Chris Aniszczyk, Cloud Native Computing Foundation; Jeff McAffer, Microsoft; Will Norris, Google; and Andrew Spyker, Netflix Using Open Source Code by Ibrahim Haddad, Samsung Research America Participating in Open Source Communities by Stormy Peters, Red Hat; and Nithya Ruff, Comcast Recruiting Open Source Developers by Guy Martin, Autodesk; Jeff Osier-Mixon, Intel Corporation; Nithya Ruff; and Gil Yehuda, Oath Measuring Your Open Source Program’s Success by Christine Abernathy, Facebook; Chris Aniszczyk; Joe Beda, Heptio; Sarah Novotny, Google; and Gil Yehuda

    The translated guides were launched at the LinuxCon + ContainerCon + CloudOpen China conference in Beijing, where The Linux Foundation also welcomed Chinese Internet giant Tencent as a Platinum Member.

    The post Open Source Guides for the Enterprise Now Available in Chinese appeared first on The Linux Foundation.

    https://www.linuxfoundation.org/blog/open-source-guides-for-the-enterprise-now-available-in-chinese/

    read more
});