NOAA’s Mission Toward Open Data Sharing



  • Open Data

    NOAA is working to make all of its data available to an even wider group of people and make it more easily understood (Image: NOAA).

    The goal of the National Oceanic and Atmospheric Administration (NOAA) is to put all of its data — data about weather, climate, ocean coasts, fisheries, and ecosystems – into the hands of the people who need it most. The trick is translating the hard data and making it useful to people who aren’t necessarily subject matter experts, said Edward Kearns, the NOAA’s first ever data officer, speaking at the recent Open Source Leadership Summit (OSLS).

    NOAA’s mission is similar to NASA’s in that it is science based, but “our mission is operations; to get the quality information to the American people that they need to run their businesses, to protect their lives and property, to manage their water resources, to manage their ocean resources,” said Kearns, during his talk titled “Realizing the Full Potential of NOAA’s Open Data.”

    He said that NOAA was doing Big Data long before the term was coined and that the agency has way too much of it – to the tune of 30 petabytes in its archives with another 200 petabytes of data in a working data store. Not surprisingly, NOAA officials have a hard time moving it around and managing it, Kearns said.

    Data Sharing

    NOAA is a big consumer of open source and sharing everything openly is part of the organization’s modus operandi. On a global level, “the agency has been a leader for the entire United States in trying to broker data sharing among countries,” Kearns said. One of the most successful examples has been through the United Nations, with an organization called World Meteorological Organization (WMO).

    Agency officials have a tendency to default making their products accessible in the public domain, something Kearns said he’d like to change. By adopting some modern licensing practices, he believes the NOAA could actually share even more information with the public. “The Linux Foundation has made progress on the community data license agreement. This is one the things I’d like to possibly consider adopting for our organization,’’ he added.

    One of the great success stories the NOAA has in terms of getting critical data to the public was after Hurricane Irma hit Florida in September 2017, he said.

    “As you can imagine, there were a lot of American citizens that were hungry for information and were hitting the NOAA websites very hard and data sites very hard,’’ he said. “Typically, we have a hard time keeping up with that kind of demand.” The National Hurricane Center is part of the NOAA, and the agency took the NHC’s website and put it on Amazon Cloud.

    This gave the agency the ability to handle over a billion hits a day during the peak hurricane season. But, he continued, “we are still … just starting to get into how to adopt some of these more modern technologies to do our job better.”

    Equal Access

    Now the NOAA is looking to find a way to make the data available to an even wider group of people and make it more easily understood. Those are their two biggest challenges: how to disseminate data and how to help people understand it, Kearns said.

    “We’re getting hammered every day by a lot of companies that want the data… and we have to make sure everybody’s got an equal chance of getting the data,” he said.

    This is becoming a harder job because demand is growing exponentially, he said. “Our costs are going up because we need more servers, we need more networks,” and it’s a problem due to budget constraints.

    The agency decided that partnering with industry would help facilitate the delivery of data.

    The NOAA is going into the fourth year of a deal it signed with Amazon, Microsoft, IBM, Google, and a nonprofit out of the University of the Chicago called the Open Commons Consortium (OCC), Kearns said. The agreement is that NOAA data will remain free and open and the OCC will host it at no cost to taxpayers and monetize services around the data.

    The agency is using an academic partner acting as a data broker to help it “flip this data and figure out how to drop it into all of our collaborators’ cloud platforms, and they turn it around and serve many consumers from that,” Kearns explained. “We went from a one-to-many, to a one-to-a-few, to a many model of distribution.”

    People trust NOAA’s data today because they get it from a NOAA data service, he said. Now the agency is asking them to trust the NOAA data that exists outside the federal system on a partner system.

    On AWS alone the NOAA has seen an improvement of over two times the number of people who are using the data, he said. The agency in turn, has seen a 50 percent reduction in hits on the NOAA servers.

    Google has loaded a lot of the agency’s climate data to its BigQuery data warehouse, “and they’ve been able to move petabytes of this data just in a few months, just because the data now has been loaded into a tool people are already using.”

    This “reduces that obstacle of understanding,’’ Kearns noted. “You don’t have to understand a scientific data format, you can go right into BigQuery… and do analyses.”

    Data Trust

    Being able to trust data is also an important component of any shared initiative, and through the NOAA’s Big Data Project, the agency is seeking ways of ensuring that the trust that comes with the NOAA brand is conveyed with the data, he said, so people continue to trust it as they use it.

    “We have a very proud history of this open data leadership, we’re continuing on that path, and we’re trying to see how we can amplify that,’’ Kearns said.

    NOAA officials are now wondering if the data is being made available through these modern cloud platforms will make it easier for users to create information products for themselves and their customers.

    “Of course, we’re also looking for other ways of just doing our business better,’’ he added. But they want to figure out if it makes sense to continue this experiment with its partners. That, he said, they will likely know by early next year.

    Watch the complete presentation below:

    The post NOAA’s Mission Toward Open Data Sharing appeared first on The Linux Foundation.

    https://www.linuxfoundation.org/blog/noaas-mission-toward-open-data-sharing/





Tmux Commands

screen and tmux

A comparison of the features (or more-so just a table of notes for accessing some of those features) for GNU screen and BSD-licensed tmux.

The formatting here is simple enough to understand (I would hope). ^ means ctrl+, so ^x is ctrl+x. M- means meta (generally left-alt or escape)+, so M-x is left-alt+x

It should be noted that this is no where near a full feature-set of either group. This - being a cheat-sheet - is just to point out the most very basic features to get you on the road.

Trust the developers and manpage writers more than me. This document is originally from 2009 when tmux was still new - since then both of these programs have had many updates and features added (not all of which have been dutifully noted here).

Action tmux screen
start a new session tmux OR
tmux new OR
tmux new-session
screen
re-attach a detached session tmux attach OR
tmux attach-session
screen-r
re-attach an attached session (detaching it from elsewhere) tmux attach -d OR
tmux attach-session -d
screen -dr
re-attach an attached session (keeping it attached elsewhere) tmux attach OR
tmux attach-session
screen -x
detach from currently attached session ^b d OR
^b :detach
^a ^d OR
^a :detach
rename-window to newname ^b , <newname> OR
^b :rename-window <newn>
^a A <newname>
list windows ^b w ^a w
list windows in chooseable menu ^a "
go to window # ^b # ^a #
go to last-active window ^b l ^a ^a
go to next window ^b n ^a n
go to previous window ^b p ^a p
see keybindings ^b ? ^a ?
list sessions ^b s OR
tmux ls OR
tmux list-sessions
screen -ls
toggle visual bell ^a ^g
create another window ^b c ^a c
exit current shell/window ^d ^d
split window/pane horizontally ^b " ^a S
split window/pane vertically ^b % ^a |
switch to other pane ^b o ^a <tab>
kill the current pane ^b x OR (logout/^D)
collapse the current pane/split (but leave processes running) ^a X
cycle location of panes ^b ^o
swap current pane with previous ^b {
swap current pane with next ^b }
show time ^b t
show numeric values of panes ^b q
toggle zoom-state of current pane (maximize/return current pane) ^b z
break the current pane out of its window (to form new window) ^b !
re-arrange current panels within same window (different layouts) ^b [space]
Kill the current window (and all panes within) ^b killw [target-window]
  • ?

    If you’re upset about your weight and want to lose a little more, get this…

    An overweight mother with pre-diabetes has just SHOCKED the medical community by losing an unheard of 22lbs pounds in just 13 days…

    …Without starving herself, she lost a total of 37lbs in the first month! <a href=http://webbyt.co/weightloss>-> CLICK HERE to see her Transformation Pics!</a>

    …Even without exercising, she went on to burn off 84lbs (almost a pound a day) and eliminated any sign of diabetes or any other life-ending diseases.

    And the amazing thing is…

    All she did was this D.I.Y. “carb-pairing” trick that reconditions your 3 female weight-loss hormones to drastically accelerate fat-burning while still eating the foods you love.

    Check it out for yourself…

    <a href=http://webbyt.co/weightloss>“Carb-Pairing” Melts Away 37 Pounds in Just 20 Days (WOMEN ONLY)</a>

    Enjoy!

    read more
});