«NOAA Environmental Data Management Framework NOAA is, at its foundation, an environmental information generating organization. Fundamental to ...»
NAO 212-15 and EDMC Procedural Directives are high level. More detailed implementation guidance and recommended practices are recorded in the NOAA Environmental Data Management Wiki (12).
Project-specific technical documentation is also more detailed.
NAO 212-13 (13) establishes requirements for the protection of all NOAA IT resources, including data and information.
NOAA's Guiding Enterprise Architecture Principles (14) states that NOAA data are a corporate resource to be managed appropriately throughout their life cycle, and calls for technical solutions that are applicable NOAA-wide, standardized, interoperable, and secure.
2.2.3. National or inter-agency policies and documents There are a number of US national or inter-agency policies and documents relevant to the governance of NOAA data management practices.
OMB Circular A-16 (15) "provides direction for federal agencies that produce, maintain or use spatial data either directly or indirectly in the fulfillment of their mission," and defines the National Spatial Data Infrastructure (NSDI) as "the technology, policies, standards, human resources, and related activities necessary to acquire, process, distribute, use, maintain, and preserve spatial data."
The Digital Government Strategy (16) is intended to "unlock the power of government data to spur innovation" by enabling "an increasingly mobile workforce to access high-quality digital government information and services anywhere, anytime, on any device." The Strategy directs agencies to architect systems for interoperability and openness, to modernize content-publication models, and to deliver better, device-agnostic digital services at a lower cost.
The 25 Point Implementation Plan to Reform Federal IT (6) calls for consolidation of surplus or underutilized data centers and establishes a "Cloud-first" policy for acquisition of new computing capability. The Federal Cloud Computing Strategy (17) lays out the Cloud approach in greater detail. * The draft OMB Memorandum on "Managing Government Information as an Asset throughout its Life Cycle to Promote Interoperability and Openness" † states that " management of information resources must begin at the earliest stages of the planning process, well before information is collected or created" and directs federal agencies to use open standards, to design systems for interoperability and information accessibility, and to create and maintain a data inventory. The alignment between the draft OMB Memo and this NOAA EDM Framework is nearly complete, except that the Memo covers personally-identifiable information in greater detail.
2.2.4. External Coordination NOAA is not the only organization that produces and uses environmental data. In order to maximize compatibility of NOAA observations with other data it is important that there be awareness of and coordination with external bodies regarding standards and technical approaches. Furthermore, many NOAA-sponsored observations are tied to significant national and international components and activities. NOAA programs that participate in international observing activities should, where possible, influence those international structures to align with and benefit from NOAA data management practices but might not be held to same level of compliance as purely in-house systems. Relevant
external bodies include, among others:
• World Meteorological Organization (WMO)
• Committee on Earth Observing Satellites (CEOS) * See also Appendix B: Cloud Computing of this Framework.
† Draft circulated for NOAA review the week of 2012-11-26; issuance date to be determined.
Version 1.0 13 2013-03-14 NOAA Environmental Data Management Framework
• Coordination Group for Meteorological Satellites (CGMS)
• Intergovernmental Oceanographic Commission (IOC)
• Group on Earth Observations (GEO) and USGEO
• Federal Geographic Data Committee (FGDC)
• Open Geospatial Consortium (OGC)
• International Organization for Standardization Technical Committee 211 for Geographic Information and Geomatics (ISO/TC211)
• International Committee for Information Technology Standards - Geographic Information Services (INCITS/L1)
• White House Office of Science and Technology Policy (OSTP) National Science and Technology Council (NSTC) committees and working groups
• Air Force Weather Agency (AFWA)
• US Integrated Ocean Observing System (IOOS) Program
• Selected programs in National Science Foundation (NSF), National Aeronautics and Space Administration (NASA), US Geological Survey (USGS), Environmental Protection Agency (EPA) and others.
• Federation of Earth Science Information Partners (ESIP) 2.2.5. Monitoring and Enforcement Formal authority and responsibility to enforce NOAA data management policy and directives resides with Line Office (LO) and Staff Office (SO) leadership and their designees. LO/SO representatives to the NOSC, CIO Council, and EDMC should ensure their leadership is aware of and understands NOAA policies and procedures for data management. The EDMC reports progress on implementation of Procedural Directives to NOSC and CIO Council on a periodic basis. EDMC members are expected to report implementation status for their Offices to the EDMC and to their Assistant CIOs and NOSC representatives.
2.3. Resources NOAA data cannot be adequately managed without proper resources, including personnel, budget and other supporting elements. Lack of resources is often a factor leading to data that are poorly documented, inaccessible, or improperly preserved.
2.3.1. Personnel Competent and motivated personnel are the key to proper management of environmental data. NOAA has many such individuals across the agency. Their work is more effective when they can exchange knowledge and work together. Such collaboration is supported in part by participation in the groups mentioned in Section 2.2.1. One intent of this document is to provide a conceptual framework and common understanding of their work.
Version 1.0 14 2013-03-14 NOAA Environmental Data Management Framework Significant improvements in NOAA data management cannot be made on the basis of volunteer efforts.
Employees responsible for any aspect of data management should have that role clearly stated in their performance plan, and should have the authority and means to carry out that role. Too often, activities such as creating and maintaining metadata, making data available to other users, or ensuring data are properly transmitted to an archival facility are treated as tasks that are ancillary to an employee's regular duties. These tasks typically are not included in employee performance plans, are not acknowledged as important by supervisors, and are not rewarded by the agency. Making good data management a part of NOAA's core business practices would help provide acceptance and recognition of these efforts. Data usage tracking and citation may also help (see Sections 3.2.9 and 3.3).
NOAA personnel should be informed of need for good data management principles. Relevant staff should be offered training in data management practices. Data-related knowledge of departing staff should be captured as part of exit procedures.
2.3.2. Budget The cost of producing observations is typically much greater than the cost of properly managing the resulting data. Satellites, radars, ship and aircraft time, and field campaigns are expensive and laborintensive, and without proper planning may consume the entire project budget while leaving little for proper data management. The Data Management Planning Procedural Directive (7) is intended in part to address this problem. Data-producing projects are required to consider how they will store, transmit, document and archive their data. Program managers, project leaders, and technical personnel should work together to adequately plan and budget for data management. (See also Section 0.) With constrained budgets, NOAA cannot improve everything at once. Therefore, the following approach
• Build new systems right the first time.
• Take advantage of tech refresh points to improve existing systems.
• Bring existing high-value datasets and systems into compliance over time, prioritizing key datasets such as those from NOAA Observing Systems of Record or those used in the National Climate Assessment.
2.3.3. Other Resources Other resources include Data Centers, pilot projects, teams, conferences, documentation, and software.
Some examples are listed below.
• Data Centers: The NOAA National Data Centers (NCDC, NGDC, and NODC) are among the world's premier facilities for long-term preservation and stewardship of environmental data.
NOAA projects, guided by the Procedure for Scientific Records Appraisal and Archive Approval (8), can work with these facilities to ensure their data are properly archived. Each Data Center is also establishing a catalog service to enable discovery of its holdings.
Version 1.0 15 2013-03-14 NOAA Environmental Data Management Framework
• Pilot Projects: Pilot projects are designed to test the implementation of new technologies prior to operational adoption. Examples include the NOAA National Data Centers Cloud Pilot * and the NOS Shared Hosting capability † to allow projects to host datasets for public access without needing to operate their own servers.
• Teams: Various cross-NOAA groups of personnel involved in EDM activities provide mutual support and guidance. The Data Management Integration Team (DMIT) is one such group with a mailing list and monthly telecons. More broadly, the Federation of Earth Science Information Partners (ESIP) ‡ is an open networked community, originally founded by NASA and NOAA, that brings together practitioners in science, data management, and IT.
• Conferences: The EDMC organizes an annual NOAA Environmental Data Management Conference for agency-wide exchange of relevant knowledge, successes, and problems. This event is typically held in June in the Silver Spring, Maryland area. Other workshops and meetings occur throughout the year.
• Documentation: The NOAA EDM Wiki (12) includes recommended practices and other guidance.
This resource is publicly readable, and can be edited by NOAA personnel who request an account. NOAA projects are encouraged to consult and contribute to this Wiki. The Wiki also includes a repository of EDM Plans § submitted in compliance with the Data Management Planning Procedural Directive.
• Software: Good data management does not necessarily require writing new code. Open-source and commercial software packages exist for editing metadata or providing user-facing (public) services for data discovery, access, or visualization. Some recommended software is listed on the EDM Wiki.
2.4. Standards Different types of standards are applicable in various phases of the Data Lifecycle. These include common vocabularies, standards for data quality, metadata standards that specify the content and structure of documentation about a dataset, data models and format standards that specify the content and structure of the digital data itself, and interface standards that specify how services are invoked.
Some standards are general-purpose and may require specialization for particular data types. Adoption of common standards supports interoperability, which enables diverse data, tools, systems, and archives to be combined without writing custom software to handle every data link. The broad use of a small set of common data, metadata, and protocol standards across NOAA, especially using international standards where possible, will decrease the cost of making and using NOAA observations, enhance the utility of the data, and help avoid redundant technical development. Existing data exchange agreements * https://www.nosc.noaa.gov/EDMC/documents/edmcon/2012_breakout_sessions/CaseyCLASS_Cloud_Access_pilot_16May2012.pdf † https://sites.google.com/a/noaa.gov/noaa-open-source-gis/noaa-hpcc-shared-hosting ‡ http://esipfed.org/ § https://geo-ide.noaa.gov/wiki/index.php?title=Category:Data_Management_Plans