ACCESS Integration Roadmaps Documentation
Integration Roadmap Tracks
Cloud Roadmaps
ACCESS Allocated Production Cloud
Infrastructure Integration Roadmap Description
Infrastructure Type(s)
This roadmap integrates a cloud resource. Researchers interact with cloud resources by provisioning virtual machines, containers, and/or storage spaces.
Summary
The ACCESS program (https://access-ci.org) enables researchers and educators to gain access to advanced computing, visualization, and data resources to accomplish their research or classroom objectives using an allocations process described at https://allocations.access-ci.org/.
This roadmap details the tasks that a resource provider must perform to make a specific cloud resource ACCESS allocated in production. These tasks cover all aspects of integration in production, including coordination activities, technical integration tasks, and ongoing support activities.
Operator Information
Operators planning to integrate a cloud resource into ACCESS with this roadmap should contact ACCESS Operations to start the integration process. ACCESS Operations will assign a Concierge Integration Expert to provide personalized hands-on assistance with the integration process, schedule an online meeting between resource provider staff and ACCESS staff to launch the integration process, and provide instructions on how to track integration progress.
Operators must perform the Required Tasks below and may perform the Optional Tasks below.
See related Roadmap Task Timeline.
Support Information
Submit a Request: Resource providers needing assistance with integration tasks can submit an ACCESS Integration and Operation Support Request using instructions on this page. Pick an “ACCESS Operational Support Issues” that best matches your request, or the catch-all issue type “ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps“.
Slack Discussion: ACCESS has a Slack channel for infrastructure operators and ACCESS to discuss integration topics. For access submit a request to be invited to the RP-ACCESS #integration-roadmaps Slack channel.
By Weekly Meetings: ACCESS holds a Zoom meeting every second Friday from 11:00 AM to Noon Central for infrastructure operators and ACCESS to discuss integration topics. To obtain Zoom coordinates submit a request for the RP and ACCESS Roadmaps Zoom meeting coordinates.
Integration Consultants: ACCESS allocated resource providers and some other types of infrastructure operators have a Concierge Integration Expert consultant available for one-on-one integration support.
Documentation:
Planning phase
Coordination: ACCESS Allocated Resource Integration Coordination v1
Coordination: Infrastructure Description v2
Technology: Cybersecurity Requirements for RPs v1
Technology: Data and Network Integration v1
Integration phase
Coordination: ACCESS Allocation Policies v1
Coordination: Knowledge Base v1
Coordination: RP Forum Participation v1
Coordination: Cybersecurity Governance Council Participation v1
Technology: Resource Metrics Data Availability Assessment v1
(Optional) Technology: ACCESS DNS Entries v1
(Optional) Technology: Local Service ACCESS IAM Integration v1
Operations phase
Support: Ticket Handling v2
Support: Operational Status Communications v1
(Optional) Support: Request RP or Site Staff Allocation v1
Technology: AMIE and Usage Reporting v1
Technology: Performance Data reporting v1
Document Management
Status: Official
Official date: 04/24/2023 04/10/2023
Coordinators: Chris Martin, ACCESS Operations
Last revised date: 08/22/2023
Retired date:
Compute Roadmaps
ACCESS Allocated Production Compute
Infrastructure Integration Roadmap Description
Infrastructure Type(s)
This roadmap integrates a high-performance computing “HPC” cluster (https://en.wikipedia.org/wiki/High-performance_computing). Researchers typically use HPC clusters by logging in to front-end nodes where they install application software and run jobs under a batch scheduler. Science gateways, workflow engines and other software tools may perform these actions on the researchers’ behalf. Jobs may run entirely in the background reading input data and generating output data, may use interactive or steered input, and may generate live or streamed visual output.
Summary
The ACCESS program (https://access-ci.org) enables researchers and educators to gain access to advanced computing, visualization, and data resources to accomplish their research or classroom objectives using an allocations process described at https://allocations.access-ci.org/.
This roadmap details the tasks that a resource provider organization must perform to make a specific HPC cluster ACCESS allocated in production. These tasks cover all aspects of integration and operations in production, including coordination, technical integration tasks, and ongoing support activities.
Operator Information
Operators planning to integrate an HPC resource into ACCESS with this roadmap should contact ACCESS Operations to start the integration process (see details in the first task below). ACCESS Operations will assign a Concierge Integration Expert to provide personalized hands-on assistance with the integration process, schedule an online meeting between resource provider staff and ACCESS staff to launch the integration process, and provide instructions on how to track integration progress.
Operators must perform the Required Tasks below and may perform the Optional Tasks below.
See related Roadmap Task Timeline.
Support Information
Submit a Request: Resource providers needing assistance with integration tasks can submit an ACCESS Integration and Operation Support Request using instructions on this page. Pick an “ACCESS Operational Support Issues” that best matches your request, or the catch-all issue type “ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps“.
Slack Discussion: ACCESS has a Slack channel for infrastructure operators and ACCESS to discuss integration topics. For access submit a request to be invited to the RP-ACCESS #integration-roadmaps Slack channel.
By Weekly Meetings: ACCESS holds a Zoom meeting every second Friday from 11:00 AM to Noon Central for infrastructure operators and ACCESS to discuss integration topics. To obtain Zoom coordinates submit a request for the RP and ACCESS Roadmaps Zoom meeting coordinates.
Integration Consultants: ACCESS allocated resource providers and some other types of infrastructure operators have a Concierge Integration Expert consultant available for one-on-one integration support.
Documentation:
Planning phase
Coordination: ACCESS Allocated Resource Integration Coordination v1
Coordination: Infrastructure Description v2
Technology: Cybersecurity Requirements for RPs v1
Technology: Data and Network Integration v1
Integration phase
Coordination: ACCESS Allocation Policies v1
Coordination: Knowledge Base v1
Coordination: RP Forum Participation v1
Coordination: Cybersecurity Governance Council Participation v1
Technology: Resource Metrics Data Availability Assessment v1
(Optional) Technology: ACCESS DNS Entries v1 (NEW)
(Optional) Technology: Local Service ACCESS IAM Integration v1
(Optional) Technology: ACCESS OnDemand Portal Integration v1
Operations phase
Technology: Deploy Globus Endpoint v1
Technology: Publish Dynamic Resource Information v2
Support: Ticket Handling v2
Support: Operational Status Communications v1
(Optional) Support: Request RP or Site Staff Allocation v1
Technology: AMIE and Usage Reporting v1
Technology: Performance Data reporting v1
Document Management
Status: Official
Official date: 04/24/2023
Coordinators: Jess Haney, ACCESS Operations
Last revised date: 08/22/2023
Retired date:
Storage Roadmaps
ACCESS Allocated Production Storage
Infrastructure Integration Roadmap Description
Infrastructure Type(s)
This roadmap integrates a storage resource. Researchers interact with storage by writing and reading data files from storage via storage interfaces.
Summary
The ACCESS program (https://access-ci.org) enables researchers and educators to gain access to advanced computing, visualization, and data resources to accomplish their research or classroom objectives using an allocations process described at https://allocations.access-ci.org/.
This roadmap details the tasks that a resource provider must perform to make a specific storage system ACCESS allocated in production. These tasks cover all aspects of integration in production, including coordination activities, technical integration tasks, and ongoing support activities.
Operator Information
Operators planning to integrate storage into ACCESS with this roadmap should contact ACCESS Operations to start the integration process. ACCESS Operations will assign a Concierge Integration Expert to provide personalized hands-on assistance with the integration process, schedule an online meeting between resource provider staff and ACCESS staff to launch the integration process, and provide instructions on how to track integration progress.
Operators must perform the Required Tasks below and may perform the Optional Tasks below.
See related Roadmap Task Timeline.
Support Information
Submit a Request: Resource providers needing assistance with integration tasks can submit an ACCESS Integration and Operation Support Request using instructions on this page. Pick an “ACCESS Operational Support Issues” that best matches your request, or the catch-all issue type “ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps“.
Slack Discussion: ACCESS has a Slack channel for infrastructure operators and ACCESS to discuss integration topics. For access submit a request to be invited to the RP-ACCESS #integration-roadmaps Slack channel.
By Weekly Meetings: ACCESS holds a Zoom meeting every second Friday from 11:00 AM to Noon Central for infrastructure operators and ACCESS to discuss integration topics. To obtain Zoom coordinates submit a request for the RP and ACCESS Roadmaps Zoom meeting coordinates.
Integration Consultants: ACCESS allocated resource providers and some other types of infrastructure operators have a Concierge Integration Expert consultant available for one-on-one integration support.
Documentation:
Planning phase
Coordination: ACCESS Allocated Resource Integration Coordination v1
Coordination: Infrastructure Description v2
Technology: Cybersecurity Requirements for RPs v1
Technology: Data and Network Integration v1
Integration phase
Coordination: ACCESS Allocation Policies v1
Coordination: Knowledge Base v1
Coordination: RP Forum Participation v1
Coordination: Cybersecurity Governance Council Participation v1
Technology: Resource Metrics Data Availability Assessment v1
(Optional) Technology: ACCESS DNS Entries v1
(Optional) Technology: Local Service ACCESS IAM Integration v1
Operations phase
Technology: Deploy Globus Endpoint v1
Support: Ticket Handling v2
Support: Operational Status Communications v1
(Optional) Support: Request RP or Site Staff Allocation v1
Technology: AMIE and Usage Reporting v1
Technology: Performance Data reporting v1
Document Management
Status: Official
Official date: 04/24/2023 04/10/2023
Coordinators: Lee Liming, ACCESS Operations
Last revised date: 08/22/2023
Retired date:
Science Gateway Roadmaps
ACCESS Integrated Science Gateway
Infrastructure Integration Roadmap Description
Infrastructure Type(s)
A science portal or science gateway is a community-developed set of tools, applications, and data integrated through a web-based portal or a suite of applications. They provide access to tools used in cutting-edge research – telescopes, seismic shake tables, supercomputers, sky surveys, undersea sensors, and more. These gateways often connect diverse resources and make them easily accessible, lowering the barriers traditionally required to access these resources.
This roadmap integrates a science gateway service (https://en.wikipedia.org/wiki/Science_gateway). Science gateways provide web-based interfaces that link advanced cyberinfrastructure (CI) components to enhance usability. The CI components linked by a science gateway differ based on specific user community needs, but can consist of one or more of the following components: High performance computing (HPC), data management, cloud-based resources, high throughput computing (HTC), user management, and authentication, authorization, and other security services.
Summary
The ACCESS program (https://access-ci.org) enables researchers and educators to gain access to advanced computing, visualization, and data resources to accomplish their research or classroom objectives using an allocations process described at https://allocations.access-ci.org/.
This roadmap details the tasks that a gateway providing organization must perform to make their gateway operational within the ACCESS environment. These tasks cover all aspects of integration and production operations of the gateway, including coordination, technical integration tasks, and ongoing support activities.
If you are interested in seeing more support directly on your science gateway, you can contact the Science Gateways Center of Excellence for technical support, usability consulting, and other community support. Apply for support at https://sciencegateways.org/work-with-us.
Operator Information
Operators planning to integrate a science gateway into ACCESS with this roadmap should contact ACCESS Operations to start the integration process (see details in the first task below). ACCESS Operations will assign a Concierge Integration Expert to provide personalized hands-on assistance with the integration process, schedule an online meeting between the gateway provider staff and ACCESS staff to launch the integration process, and provide instructions on how to track the integration progress.
Operators must perform the Required Tasks below and may perform the Optional Tasks below.
Support Information
Submit a Request: Science gateway providers needing assistance with integration tasks can submit an ACCESS Integration and Operation Support Request using instructions on this page. Pick an “ACCESS Operational Support Issues” that best matches your request, or the catch-all issue type “ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps“.
Slack Discussion: ACCESS has a Slack channel for infrastructure operators and ACCESS to discuss integration topics. For access submit a request to be invited to the RP-ACCESS #integration-roadmaps Slack channel. This Slack channel will be used for ACCESS Resource Providers and ACCESS Allocated Science Gateways.
By Weekly Meetings: ACCESS holds a Zoom meeting every second Friday from 11:00 AM to Noon Central for infrastructure operators and ACCESS to discuss integration topics. To obtain Zoom coordinates submit a request for the RP and ACCESS Roadmaps Zoom meeting coordinates. This meeting will be used for ACCESS Resource Providers and ACCESS Allocated Science Gateways.
Integration Consultants: ACCESS integrated science gateway providers have a Concierge Integration Expert consultant available for one-on-one integration support.
General Science Gateway Consultants: Seeking technical advice, direct access to a gateway developer, a usability review of your gateway, or sustainability consulting? You can receive these and more by requesting help from the Science Gateways Center of Excellence (SGX3). Apply for support at https://sciencegateways.org/work-with-us.
Documentation:
Introduction to Roadmaps slides
Planning phase
Coordination: ACCESS Science Gateway Integration Coordination v1
Integration/Operations phase(s)
Online Services Roadmaps
ACCESS Production Online Service
Infrastructure Integration Roadmap Description
Infrastructure Type(s)
This roadmap integrates online services provided by ACCESS projects and their vendors / partners into the ACCESS operational environment. The roadmap is intended for online services that are visible outside a single ACCESS project and used by other ACCESS projects, resource providers, developers, or ACCESS users. Online services are accessed using a variety of network protocols suited for their function and target audience. Examples include ACCESS websites; Atlassian, GitHub, Google, and Globus services; DNS services; IAM services, and various APIs.
Summary
This roadmap enumerates the tasks for integrating online services into the ACCESS environment for use by ACCESS users, resource providers, ACCESS projects, Science Gateways and/or other developers. These tasks cover all aspects of integration and operations, including coordination, technical integration, and ongoing support activities. Optional tasks detail ways to integrate that are available but not required.
Operator Information
This roadmap was defined by ACCESS projects for use primarily by ACCESS projects. ACCESS projects participate in the roadmap development process and should know how to follow this integration roadmap without a hands on concierge. Operators must perform the Required Tasks below and may perform the Optional Tasks below.
See related Roadmap Task Timeline.
Support Information
Submit a Request: Operators needing assistance with integration tasks can submit an ACCESS Integration and Operation Support Request using instructions on this page. Pick an “ACCESS Operational Support Issues” that best matches your request, or the catch-all issue type “ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps“.
Slack Discussion: ACCESS projects have a Slack channel for discussing the development and application of integration roadmaps. For access, talk to your projects’ participant in the Integration Roadmaps Working Group, or submit a request to be invited to the ACCESS-CI #wg-integration-roadmaps Slack channel.
Documentation:
Introduction to Roadmaps slides
Integration Tasks
(Optional) ACCESS DNS Entries
(Optional) Local Service ACCESS IAM Integration
(Optional) Request RP or Site Staff Allocation
(Optional) Performance Data reporting
None of the above look right for you?
If you need to integrate a type of cyberinfrastructure resource with ACCESS that isn’t listed above, consider the following option(s):
Make a New ACCESS Integration Roadmap
If you want to integrate a cyberinfrastructure resource into the ACCESS ecosystem and no existing integration roadmap seems appropriate for your resources, you may want to propose a new ACCESS integration roadmap. This page explains the benefits of an integration roadmap and how to create a new one. At the end of this page, you’ll find a link to begin the process of creating a new integration roadmap, but please review the information here first.
What does it mean to integrate a cyberinfrastructure resource with ACCESS? For most resources, integration with ACCESS means participating in one or more of ACCESS’s coordination services. These coordination services (listed below) make the ACCESS ecosystem—including the individual resources within it—easier for researchers to understand and use.
An ACCESS integration roadmap details the ACCESS coordination services most appropriate for a specific kind of cyberinfastructure resource and the steps a resource provider must take to participate in them.
Steps for Creating a New ACCESS Integration Roadmap
The rest of this guide will fill in the details, but here’s an overview of the steps for creating a new ACCESS Integration Roadmap.
Inform the ACCESS Infrastructure Integration Steering Committee (IISC) that you are beginning to work on an integration roadmap for a new kind of ACCESS-integrated cyberinfrastucture resource.
Form a working group to help you identify the necessary pieces of the integration roadmap.
Draft the integration roadmap, by copying tasks from other roadmaps and defining tasks specific to the new type of resource.
Publish the new integration roadmap in ACCESS’s integration roadmap library.
Announce the new integration roadmap.
Why Create a New ACCESS Integration Roadmap?
If you’re adding the first resource of a type that has never been integrated with ACCESS before, there won’t be an existing integration roadmap for you to follow. Creating a new roadmap offers the following benefits.
It will help you take advantage of the coordination services ACCESS has to offer. Once you decide which services will add value to your resource, the roadmap can help you keep track of your progress as you work with each coordination service.
It will give other resource providers a sensible roadmap for adding their resources to ACCESS.
As more resources like yours are added to ACCESS, researchers using these resources will benefit from appropriate commonalities provided by the integration roadmap. ACCESS-integrated resources will work in similar ways when it’s helpful.
If it isn’t likely there will ever be other resources like yours in ACCESS, you can use the information below to plan your integration without actually creating an integration roadmap.
ACCESS’s Coordination Services
The major coordination services ACCESS currently offers are listed below. Each service provides benefits to both researchers and resource providers. You should review this list and decide which of these services would help you achieve your goals for ACCESS integration.
ACCESS Resource Discovery: An ACCESS-wide searchable/browseable directory of resources. Having a resource listed in ACCESS’s resource directory provides greater visibility for the resource, helping resource providers recruit new users. Researchers have a convenient place to look for resources they can use to carry out their research projects.
ACCESS Allocations: A system for requesting permission to use resources in support of research projects. Resource providers can use this to recruit new and returning users for their resources. Researchers use this to gain access to additional resources for their research projects.
CONECTnet: Wide area network for efficient and resilient data transfer and novel network applications across the ACCESS ecosystem. Resource providers benefit from a high-performance, resilient network connection for their resources. Researchers benefit from advanced network connectivity between ACCESS resources.
ACCESS Coordinated User Support: ACCESS-wide services for providing technical support to researchers. Researchers benefit from a single place to request help when using any ACCESS resource. Resource providers benefit from being able to collaborate with other ACCESS personnel on user support when issues involve ACCESS services or another ACCESS resource.
ACCESS Metrics and Monitoring: An ACCESS-wide service for gathering and presenting resource usage metrics. Researchers have a single place to go to find out how much they’ve used of ACCESS resources and compare it to overall use. Resource providers get a standardized way to report use of their resources to sponsors and other stakeholders.
ACCESS Single Sign-On: ACCESS-wide user identities and authentication service. Reseachers have a single ID to use with all ACCESS resources and single sign-on across ACCESS websites and resources. Resource providers can use ACCESS IDs and authentication rather than maintaining local accounts and user credentials (passwords, keys).
Inform the ACCESS Infrastructure Integration Steering Committee (IISC)
After you’ve reviewed this guide and decided that you want to propose a new integration roadmap, the first step is to inform the Infrastructure Integration Steering Committee (IISC). The last section of this guide provides the link you can follow to formally notify the IISC.
The IISC includes representatives of ACCESS’s operations team and project management. Their main purpose is to make sure you have the resources you need to accomplish your objective. These resources include: ACCESS project contacts and introductions, documentation, and access to our shared Google Drive. A liaison from the IISC will contact you to review your plans and offer assistance. After meeting with you, the liaison will inform the ACCESS Executive Council (EC) about your plan so they can prepare their project areas to provide the resources you’ll need.
Form a Working Group
When creating a new ACCESS integration roadmap, it’s important to get input from ACCESS service providers and—if possible—from other resource providers who are likely to follow the roadmap. A good way to get that input is to form a working group. ACCESS working groups allow stakeholders from the ACCESS projects and resources provider organizations to meet regularly to work together on a common goal. For example, you can launch a working group to create a new integration roadmap, invite stakeholders from the relevant areas in ACCESS, and when the roadmap is completed, the working group ends.
Creating a working group is easy. The first step is to draft a charter. The charter defines the working group and has the following elements.
A name for the working group
An initial chairperson (two chairpersons is also fine)
A one-paragraph statement of the working group’s purpose
A list of specific goals
A list of stakeholders
A tentative timeline for deliverables
Stakeholders are people who have or may have an interest in the outcome of the working group or who have expertise that the working group needs. For example, you will likely need representatives from the major ACCESS project areas and new or prospective ACCESS resource providers. Deliverables are concrete things the working group will produce, such as a new integration roadmap.
When you have your charter document, review it with your project leadership and make certain they approve of you working on it. Then, send a note to your IISC liaison (see previous section) so he or she can inform the ACCESS Executive Committee of the new working group. (The Executive Committee needs to know because you’ll likely need representatives from their projects to participate as stakeholders in the working group.)
Once your charter is approved, create a new folder for the working group within the ACCESS Working Groups Google Drive folder. Add the charter document. Review the ACCESS cross-track technical coordination schedule to find a good time for the first meeting and ask on the ACCESS-CI slack’s #access-wide channel if anyone is using the two most-likely options, just in case. Create a new document in the working group Google Drive folder for meeting agendas and notes and include the date, time, and coordinates for the first meeting. (It’s helpful to keep all meeting notes in a single document so they can be referenced during meetings.)
Finally, draft an announcement for the working group that references the charter, key stakeholders who you’d like to have participate, agenda for the first meeting, and meeting coordinates. Post the announcement on the ACCESS-CI slack’s #access-wide channel and the ACCESS-RP Communications slack’s #general channel. If you need participation by current resource providers, ask the ACCESS Operations team to email your announcement to all current resource provider contacts. Now you’re ready for your first working group meeting!
What to Put In an Integration Roadmap
The audience of an ACCESS integration roadmap is a resource provider who needs to integrate a new resource into the ACCESS ecosystem. Roadmaps consist of a series of tasks that the resource provider must accomplish in order to set things up with the appropriate ACCESS coordination services. For the technical details on how to construct the files needed for an integration roadmap and its tasks in GitHub, please see the Contributing guide.
Most integration roadmaps have two types of tasks:
Establishing personnel contacts
Configuring and documenting software/services
In both cases, you’ll need the ACCESS project representatives in your working group to tell you what they need resource providers to do. This should be the main topic in your working group meetings. Once you’ve described your resource and what makes it different from the current resource types, the ACCESS area representatives should be able to pinpoint tasks from other integration roadmaps that will be the same in this new roadmap (and can be copied), tasks from other roadmaps that don’t seem applicable (and can be left out), and any new tasks that are needed for this type of resource that haven’t been necessary in other roadmaps.
In regard to personnel contacts, ACCESS areas typically need the resource provider to provide a list of contacts for specific functions. It’ll usually be a combination of management personnel (PIs, project managers) and technical personnel (HPC system adminstrators, network administrators, storage/data specialists, security personnel). Once all of the ACCESS representatives in your working group have given you the contacts they need, consolidate them into a single list and create a task at the beginning of the roadmap so the resource provider can just fill out a single form with all of the necessary contact information.
For configuring and documenting software and services, each ACCESS project area will have its own list of ways resource providers can engage with their area. It’s very likely these will be similar to (or even the same as) tasks in existing integration roadmaps, so the working group representatives may only have to point to existing tasks in other roadmaps and ask to have them included in your new roadmap. If the tasks don’t need to be customized, you can include them in your roadmap by reference instead of making new copies. (Just be aware that if the task is edited for another roadmap, those edits will also appear in your roadmap.) If a task needs to be customized for this type of resource, or if you don’t want to take the chance that edits to the task in other roadmaps also change the task in your roadmap, make a copy. But if you make a copy, be aware that any improvements to the task’s text in other roadmaps won’t be reflected in yours.
Get an Integration Roadmap Reviewed and Approved
Assuming you have the right stakeholders represented in your working group, you shouldn’t need additional reviews or approvals once the tasks have been finalized because everyone who is involved in the roadmap will have been engaged already. Feedback from new resource providers won’t likely appear until your roadmap has been published and announced. You can proceed directly to publishing and announcing your roadmap!
Publish and Maintain an Integration Roadmap
As outlined in the Contributing guide, your new roadmap will consist of a pull request against the ACCESS Integration Roadmaps GitHub repository. When you submit the pull request, it will be reviewed by the ACCESS Operations team’s Resource Integration specialists. The goal of this review will be to ensure that the ACCESS Operation team can support resource providers in carrying out the tasks in your roadmap. If they have any questions or concerns, they will respond to the pull request with a list of suggestions or requests. Once the pull request is approved and merged, your roadmap will be available to the ACCESS community!
As new resource providers use your integration roadmap, the ACCESS Operations team will collect feedback in support tickets and other channels. If changes are needed, they can be made through the process outlined in the Contributing guide.
Announce a New Integration Roadmap
Your new ACCESS integration roadmap will only be useful if people know about it! We recommend the following ways to let relevant people know about your roadmap.
Post a link to it, including a brief blurb about what types of resources it’s for, on the ACCESS-CI slack server’s #access-wide channel.
Post the same announcement on the RP-ACCESS Communications slack server’s #general channel.
Post a news item on the ACCESS Infrastructure News publishing service. This will result in a news story in ACCESS’s staff newsletter and one or more news channels for resource providers and other community members.
Get Started
If you’re ready to get started on a new ACCESS integration roadmap, the first step is to alert the ACCESS Infrastructure Integration Steering Committee that you’re ready to get started. We use GitHub to coordinate this work, so please follow this link and then press the green “Get started” button next to “New ACCESS integration roadmap.” Enter the requested information and then press “Submit new issue.” The steering committee members will be alerted and you’ll be contacted soon to discuss your plans.
ACCESS Integration Roadmaps Tasks
Note
These are all Roadmap Tasks which are temporarily placed here to stop build failures for files not referenced in toctree(s) appropriately.
ACCESS Affinity Groups and Science Gateways
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Operations
Operator role(s): Gateway administrator and contacts
Summary
ACCESS uses affinity groups to enable communications among members of communities. Science gateway providers should join the science gateway affinity group and the affinity groups of any resource providers that the science gateway uses.
Prerequisite tasks
The gateway should have completed registration. See “Science Gateway Registration”.
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
See https://support.access-ci.org/affinity-groups for a listing of groups and steps for joining. You must be logged in to your ACCESS account to access these groups.
A Science Gateway Affinity Group has been requested but may not appear on this site as of the publishing of this document.
Resource providers use affinity groups to communicate with their users, including science gateway providers. The science gateway provider may wish to communicate this information to their users through their own communication channels.
ACCESS science gateway community members and ACCESS staff communicate using the science gateways affinity group. This may for example include changes to policies, new services, and informal support.
Some resource providers may have additional communication channels.
ACCESS Allocated Resource Integration Coordination v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Planning
Complete by phase: ongoing
RP role(s): PI and co-PI(s), Resource integration coordinator(s)
Summary
A resource provider wishing to integrate an ACCESS allocated compute, storage, or cloud resource contacts ACCESS to start the integration process, provides basic resource information, and identifies resource provider staff contacts that will be involved in coordination, technical integration, and ongoing support activities.
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Prerequisite tasks
None
Operator Instructions
Integration Launch
The integration engagement and process begins when the resource PI/co-PIs and an RP designated Resource Integration Coordinator communicates with ACCESS Operations, identifying themselves and their PI, co-PI, and coordinator roles, provides their institution name, the name and type of the resource they plan to integrate, and which ACCESS integration roadmap they plan to follow (if known). The RP may communicate this information by submitting an ACCESS ticket using the Integration and Operations Requests, or by contacting the ACCESS Resource Provider Coordinator directly.
In response, ACCESS will identify an Integration Concierge, and if needed schedule a call between the Resource Integration Coordinator, anyone else the RP wishes to invite, the ACCESS Integration Concierge, and the ACCESS Resource Provider Coordinator. The purpose of the meeting is to introduce RP and ACCESS staff to each other, to discuss the integration process and timeline, and to introduce integration roadmap documentation. A call may not be necessary if the RP is already familiar with the ACCESS integration process.
The Integration Concierge acts as the resource or service operator’s primary point of contact for all questions and issues in selecting and completing the integration roadmap.
The Resource Integration Coordinator is responsible for coordinating all RP activities, for identifying and maintaining the list of RP contacts involved in different aspects of coordination, integration, and resource support (see the next section), for communicating relevant ACCESS integration information and Roadmap based resource integration document to RP staff, and for managing RP staff access to ACCESS services.
Identity Resource Provider Contacts
Soon after the Integration Launch the Resource Integration Coordinator must provide each of the following RP contacts:
Public relations and media contact(s)
RP Forum participant(s)
Allocations process contact(s)
AMIE technical contact(s)
Documentation and knowledge base contact(s)
Researcher support contact(s)
System administrator(s)
Cybersecurity and incident response contact(s)
Data and networking contact(s)
Resource news and outages publisher(s)
Metrics and performance data contact(s)
If the RP doesn’t know who some of these contacts will be, or wishes not to identify them until relevant integration effort ramps up, please name the Integration Coordinator as the placeholder for those roles. In other words, ACCESS needs each someone to be the contact in each of the above areas, even if it’s the coordinator who will eventually hand off effort to someone else. As integration activities ramps up the Integration Coordinator can add other RP staff replacing themselves as necessary.
The Integration Coordinator should enter and update their RP contacts in the resource specific tab here:
The Integration Coordinator is responsible for maintaining accurate contact information in this spreadsheet and updating when RP staff changes. We recommend that the Integration Coordinator review and correct their contacts every 6 months. This sub-task should take ~1 hour to complete initially, and ~½ hour annually to keep up-to-date.
ACCESS Integration Roadmaps task lists which RP staff contacts normally perform that task.
Document Management
Status: Official
Official date: 4/24/2023 2/1/2023
Coordinators: JP Navarro, ACCESS Operations
Last revised date: 1/17/2023
Retired date:
ACCESS Allocation Policies v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Planning
Complete by phase: Ongoing
RP role(s): Allocations process contact(s)
Summary
Resource Providers will need to have proper representation for Allocation questions, reviewing RP requests and provide guidance to both ACCESS Allocations staff and review panels.
Prerequisite tasks
None
Support Information
The ACCESS Allocations staff can be contacted through the following mechanisms:
-
For requesting assistance and access to resources listed above
-
For general discussion with ACCESS staff and fellow RPs
-
For recommendations, requests, and feedback pertaining to ACCESS Allocation Services
Detailed Instructions
RPs who are designed as Allocations Contacts, who are responsible for reviewing requests for their resources and ensuring their site’s documentation is up to date, are encouraged to utilize the following tools & resources to assist with their involvement in the Allocations process. Feedback related to this document (and ACCESS Allocation Services as a whole) is welcomed through our feedback form. If you’re a new RP, please familiarize yourself with the tools and processes mentioned herein and reach out to Allocations staff for assistance.
XRAS (eXtensible Resource Allocation Service)
The main tool under constant development and refinement for ACCESS Allocations. The software suite provides distinct interfaces for Submit, Review, and Administration components required for end-to-end services
-
Serves as the mechanism to review requests for your resource(s). We recommend that you bookmark this page!
-
Documents the allocations cycle and review guidelines. We recommend that you regularly refer to this documentation as you complete any and all review assignments
-
In order to create an open, inviting, and democratized allocations marketplace various training and reference materials are provided
-
Provides overviews of a user’s or a project’s history of actions
To request access to this tool, please create a ticket
Available Allocation Opportunities
Details the four opportunities to request allocations on ACCESS resources
-
Allocations for the first three opportunity levels are awarded in ACCESS Credits
A credit exchange rate must be established by the Resource Provider for the systems they will allocate. This is done through the CyberInfrastructure Description Repository (CiDeR)
RPs can contact Allocations if they have questions on setting their exchange rate
Tip for storage resource providers: At this time, most ACCESS storage resources have set an exchange rate of one ACCESS credit to one resource unit. Additionally, most storage resources have a rate of one resource unit to 1 GB (gigabyte) of storage. Keeping these consistent, if possible, will help reduce confusion (and avoid support tickets). If you intend to set different rates for your resource, please review the implications with a representative from the ACCESS Allocations team.
-
Summarizes all resources currently available in the ACCESS ecosystem
The RP Allocation contact is responsible for contributing and reviewing the descriptions and associated content for each allocated resource (also through CiDeR)
Other Helpful Resources
Some additional links that may be of benefit to RPs:
-
Post upcoming events that the ACCESS ecosystem could benefit from
-
For your site’s resources
Thank you for providing your essential expertise and assistance to the ACCESS ecosystem!
Operating Instructions
RPs who are designed as Allocations Contacts, who are responsible for reviewing requests for their resources and ensuring their site’s documentation is up to date, are encouraged to utilize the following tools & resources to assist with their involvement in the Allocations process. Feedback related to this document (and ACCESS Allocation Services as a whole) is welcomed through our feedback form. If you’re a new RP, please familiarize yourself with the tools and processes mentioned herein and reach out to Allocations staff for assistance.
XRAS (eXtensible Resource Allocation Service)
The main tool under constant development and refinement for ACCESS Allocations. The software suite provides distinct interfaces for Submit, Review, and Administration components required for end-to-end services
-
Serves as the mechanism to review requests for your resource(s). We recommend that you bookmark this page!
-
Documents the allocations cycle and review guidelines. We recommend that you regularly refer to this documentation as you complete any and all review assignments
-
In order to create an open, inviting, and democratized allocations marketplace various training and reference materials are provided
-
Provides overviews of a user’s or a project’s history of actions
To request access to this tool, please create a ticket
Available Allocation Opportunities
Details the four opportunities to request allocations on ACCESS resources
-
Allocations for the first three opportunity levels are awarded in ACCESS Credits
A credit exchange rate must be established by the Resource Provider for the systems they will allocate. This is done through the CyberInfrastructure Description Repository (CiDeR)
RPs can contact Allocations if they have questions on setting their exchange rate
Tip for storage resource providers: At this time, most ACCESS storage resources have set an exchange rate of one ACCESS credit to one resource unit. Additionally, most storage resources have a rate of one resource unit to 1 GB (gigabyte) of storage. Keeping these consistent, if possible, will help reduce confusion (and avoid support tickets). If you intend to set different rates for your resource, please review the implications with a representative from the ACCESS Allocations team.
-
Summarizes all resources currently available in the ACCESS ecosystem
The RP Allocation contact is responsible for contributing and reviewing the descriptions and associated content for each allocated resource (also through CiDeR)
Contact Information
The ACCESS Allocations staff can be contacted through the following mechanisms:
-
For requesting assistance and access to resources listed above
-
For general discussion with ACCESS staff and fellow RPs
-
For recommendations, requests, and feedback pertaining to ACCESS Allocation Services
Other Helpful Resources
Some additional links that may be of benefit to RPs:
-
Post upcoming events that the ACCESS ecosystem could benefit from
-
For your site’s resources
CIDeR (Cyberinfrastructure Description Repository)
To update resource information
Thank you for providing your essential expertise and assistance to the ACCESS ecosystem!
Document Management
Status: Official
Official date: 4/24/2023 1/23/2023
Coordinators: Ken Hackworth, ACCESS Allocations
Last revised date: 6/8/2023
Retired date:
ACCESS DNS Records v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Network, storage, and system administrators
Summary
ACCESS Operations manages the access-ci.org DNS domain and sub-domains. ACCESS projects, integrated resources, and central service may request DNS entries and DNS subdomains for their ACCESS integrated infrastructure.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
ACCESS projects, integrated resources, and central service operators may request DNS entries or DNS subdomains by submitting a ticket using a Subject that starts with “ACCESS DNS entries for …” using the Support Portal ticket form. On the ticket form please select “Not related to a resource” (since your request is to ACCESS Operations and not to a resource operator), select “No” for allocations related, and the “Operations Services” category.
ACCESS OnDemand Portal Integration V1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): System administrator(s)
Summary
This task provides Resource Providers instructions on how to deploy an ACCESS integrated OnDemand portal. This task will generally be performed by RP system administrators.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Required
Install Open OnDemand. Refer to the open ondemand installation documentation.
Register / document ACCESS OnDemand portal availability
In the Support Portal
Check to validate that your OnDemand instance is listed on the OnDemand Support Page https://support.access-ci.org/ondemand
If it is not, then click the link to Submit a Ticket. Please include the following information.
Institution Name
OnDemand Resource Name
Your Logo, with a width of about 300 pixels and the height relative to the width.
Your OnDemand Portal URL.
With Operations
Please visit https://operations.access-ci.org/pub/for_operators
Install and set up the
xdmod-ondemand-exportPython script (following its README) for sending OnDemand log data via HTTPS POST to the ACCESS Metrics team for inclusion in ACCESS XDMoD.
Optional
Integrate with ACCESS IAM Authentication.
Open OnDemand has documenatation for integrating OnDemand with ACCESS authentication.
Add the ACCESS Menu
You can optionally add a menu to Open OnDemand’s navigation bar for links back to ACCESS resources.
OSC provides a ACCESS Menu package with documentation on how to install additional packages or configure OnDemand to show this new ACCESS menu.
Document Management
Status: Final
Official date: 4/24/2023 03/16/2023
Coordinators: Jeff Ohrstrom, ACCESS Support
Last revised date: 10/18/2023
Retired date:
ACCESS Science Gateway Integration Coordination v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Planning
Complete by phase: ongoing
RP role(s): PI and co-PI(s), Science Gateway integration coordinator(s)
Summary
A science gateway wishing to integrate with ACCESS contacts ACCESS to start the integration process, provides basic science gateway information, and identifies science gateway staff contacts that will be involved in coordination, technical integration, and ongoing support activities. This coordination can begin before or after an ACCESS allocation is awarded.
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Prerequisite tasks
None
Operator Instructions
Integration Launch
The integration engagement and process begins when the science gateway PI/co-PIs and a designated Science Gateway Integration Coordinator communicates with ACCESS Operations, identifying themselves and their PI, co-PI, and coordinator roles, providing their institution name, and the name of the science gateway they plan to integrate. Communicate this information by submitting an ACCESS ticket using the Support Portal ticket form, or by contacting the ACCESS Resource Provider Coordinator directly.
In response, ACCESS will identify an Integration Concierge, and if needed schedule a call between the Resource Integration Coordinator, anyone else the RP wishes to invite, the ACCESS Integration Concierge, and the ACCESS Resource Provider Coordinator. The purpose of the meeting is to introduce RP and ACCESS staff to each other, to discuss the integration process and timeline, and to introduce integration roadmap documentation. A call may not be necessary if the RP is already familiar with the ACCESS integration process.
The Integration Concierge acts as the resource or service operator’s primary point of contact for all questions and issues in selecting and completing the integration roadmap.
The Resource Integration Coordinator is responsible for coordinating all RP activities, for identifying and maintaining the list of RP contacts involved in different aspects of coordination, integration, and resource support (see the next section), for communicating relevant ACCESS integration information and Roadmap based resource integration document to RP staff, and for managing RP staff access to ACCESS services.
Identity Resource Provider Contacts
Soon after the Integration Launch the Science Gateway Integration Coordinator must provide each of the following contacts:
Gateway administrator(s)
Cybersecurity and incident response contact(s)
If the Gateway Administrator doesn’t know who some of these contacts will be, or wishes not to identify them until relevant integration effort ramps up, please name the Integration Coordinator as the placeholder for those roles. In other words, ACCESS needs each someone to be the contact in each of the above areas, even if it’s the coordinator who will eventually hand off effort to someone else. As integration activities ramps up the Integration Coordinator can add other staff replacing themselves as necessary.
The Integration Coordinator should enter and update their contacts in the resource specific tab here:
The Integration Coordinator is responsible for maintaining accurate contact information in this spreadsheet and updating when gateway staff changes. We recommend that the Integration Coordinator review and correct their contacts every 6 months. This sub-task should take ~1 hour to complete initially, and ~½ hour annually to keep up-to-date.
Leverage Science Gateways Center of Excellence (SGX3) Services
Seeking additional support for your science gateway? Work with the Science Gateways Center of Excellence (SGX3):
Usability Consultancy: The SGX3 Usability Team can help create a new user interface or improve an existing design. Apply for a usability consultation at https://sciencegateways.org/work-with-us.
Technical Consultancy: SGX3 is here to provide advice on choosing the best platform for your project. You can also request embedded development support - a great option if you need a developer for a small period of work or need a novel expert to help integrate your science into your gateway. Request a technical consultation at https://sciencegateways.org/work-with-us.
Sustainability Training: Seeking how to sustain your science gateway financially? Learn to articulate your project’s value, define stakeholder interactions, and explore funding models at an SGX3 sustainability training workshop. Learn more at https://sciencegateways.org/education-training/sustainability-training.
SGX3 Workforce Development: Interested in receiving mentoring, career development, or participating in a hackathon for incorporating science gateways into the classroom? Discover some of our Workforce Development offerings at https://sciencegateways.org/education-training/.
Contact Us: Want to work with us? Request a consult, share a letter of collaboration, or ask a question by emailing help@sciencegateways.org.
AMIE and Usage Reporting v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Planning
Complete by phase: Operations
RP role(s): AMIE technical contact(s)
Summary
Resource Providers will implement a client for the Account Management Information Exchange (AMIE) protocol, to receive allocations information from ACCESS Allocations and report necessary information back to ACCESS.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
The Resource Provider must first identify a technical contact, who will have an initial kick-off meeting with the Allocations team to discuss the overview of AMIE implementation. The RP technical team will then follow the AMIE Documentation to create a test client. This client must be capable of responding to ACCESS allocation AMIE packets, and respond appropriately, creating or modifying the allocations information in the local site’s infrastructure. The Allocations team will be available to consult with the development team to assist with technical questions. After the test client has been created and thoroughly tested, the RP will notify the Allocations team that the AMIE implementation is ready for production. They will be given a production API key, and the AMIE site will be activated in production.
The AMIE documentation also contains information about the Usage Reporting API. RPs must create a client that will report the local usage data from their site and post this usage back to Allocations via the API.
The RP will be expected to maintain a technical contact with the Allocations team, to be able to respond to technical questions or error reports, or to adjust the client implementation as needed in the future.
Document Management
Status: Official
Official date: 4/24/2023 1/23/2023
Coordinators: Nathan Tolber & Rob Light, ACCESS Allocations
Last revised date: 1/23/2023
Retired date:
CONECTnet Integration v1
Infrastructure Integration Roadmap Task
Task Type(s): {Coordination, Technology, Support}
Start by phase: {Planning, Integration, Operations}
Complete by phase: {Planning, Integration, Operations, Ongoing}
RP role(s): Data and networking contact(s)
Summary
CONECTnet is the wide area network infrastructure and services that connect ACCESS RPs and participant sites. In most cases, the Internet2 Research and Education (R&E) network will be the WAN provider. However, due to financial or other constraints, some sites’ only connectivity option may be via a commodity Internet service provider. In either case, ACCESS CONECT network engineers are available to work with the connecting sites to help with integration questions and optimize their network performance.
CONECTnet participants are encouraged to take advantage of network measurement and test infrastructure that is available through Internet2 and also through participation in an ACCESS perfSONAR mesh.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
To integrate with CONECTnet we request that you complete a Site Survey, schedule a meeting with Data
Steps to integrate with CONECTnet:
RP network engineers fill out the CONECTnet Site Survey. This may require collaboration input from the RP’s system administrators and applications support personnel.
Submit a ACCESS ticket requesting to participate in CONECTnet, mentioning that you have completed the CONECTnet Site Survey.
The CONECTnet team will acknowledge receipt of a connection request within 3 days and schedule a follow up meeting to review the CONECTnet Site Survey and plan the next steps.
perfSONAR mesh testing is available for any site that chooses to participate (10Gb/s throughput/loss/latency)
Is this tied to CONECTnet Integration?
DNS is being provided by CONECTnet for access-ci.org
Document Management
Status: Official
Official date: 4/24/2023 03/15/2023
Coordinators: Kathy Benninger and David Wheeler, ACCESS Operatons
Last revised date: 03/09/2023
Retired date:
Cybersecurity Governance Council Participation v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Cybersecurity and incident response contact(s)
Summary
The Cybersecurity Governance Council (CGC) works in cooperation with the ACCESS Executive Council (EC) to provide the oversight of cybersecurity operations and the formation and dissemination of cybersecurity policies. These policies will encompass requirements, procedures, and guidelines for ACCESS services, infrastructure, and Resource Providers (RPs). The CGC also works to share cybersecurity information, discuss and disseminate new cybersecurity threats and vulnerabilities and exchange best practices.
Membership in the CGC will consist of a representative from each RP and Track and an ex-officio member representative from the ACCESS Coordination Office (ACO). The CGC will be convened by the EC and chaired by the ACCESS Security Manager (ASM).
Effort
Bi-weekly meetings transitioning to monthly meetings lasting 30/min to 1/hour.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Request membership to the CGC via e-mail to Derek Simmel, dsimmel@psc.edu
Participation from all members is vital to realize the security goals of ACCESS for its users and participants. As such, all CGC members agree to:
Regularly participate in the regular CGC meetings,
Promptly respond to communications about ACCESS security incidents,
Facilitate the sharing of cybersecurity information relevant to ACCESS and,
Provide input to proposed drafts of ACCESS cybersecurity policies.
There are regular meetings, held at least bi-weekly, and run by the ASM.
Additional links
Document Management
Status: Official
Official date: 4/24/2023 <mm/dd/yyyy>
Coordinators: Derek Simmel & Shane Filus, CONECT Cybersecurity Program
Last revised date: 04/16/2023
Retired date:
Cybersecurity Requirements for ACCESS Services v1
Infrastructure Integration Roadmap Task
Task Type(s): {Coordination, Technology, Support}
Start by phase: {Planning, Integration, Operations}
Complete by phase: {Planning, Integration, Operations, Ongoing}
RP role(s): Cybersecurity and incident response contact(s)
Summary
<Summarizes the purpose or goal for the task, who performs the task, and approximate effort. If this task replaces or upgrades one or more other tasks, provide references to those tasks and summarize significant new or changed elements.>
Prerequisite tasks
None
OR
<task name and link>
…
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
<REPLACE the above boilerplate sentence if you want a different operator support process for this task>
Detailed Instructions
<Detailed task instructions that may include separate sections for both one-time / setup tasks and ongoing or recurring activities.>
Cybersecurity Requirements for RPs v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Operations
RP role(s): Cybersecurity and incident response contact(s)
Summary
The cybersecurity requirements for RPs ensures that the ACCESS community’s cybersecurity needs are satisfied when a new service is added to the system. For instance: membership and participation in incident response, vulnerability patching and mitigation, retention of system logs, etc. Requirements are driven by ACCESS community policies.
The purpose of this document is to define the expectations and responsibilities of the ACCESS Resource Providers with respect to security and incident response. These requirements ensure the ability to (i) protect ACCESS assets, (ii) respond to threats to those assets, and (iii) maintain the lines of communication necessary for the former two goals.
Effort
It is difficult to estimate the time and effort required because each RPs cybersecurity program and implementations are different. However, many of these are considered best practices or baseline controls and RPs are likely to be implementing most of these already. Nonetheless, resources must be devoted to ensure that the standards are properly implemented and processes developed to ensure they are maintained.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Description
Requirements for this review are driven by ACCESS community policies, listed below.
ACCESS Training and Awareness Policy (Under Development)
ACCESS Identity and Access Management Policy (Under Development)
ACCESS Information Classification Policy (Under Development)
ACCESS Disaster Recovery Policy (Under Development)
Responsibilities
Beyond just reporting security incidents, the RPs incident response point of contact is expected to actively participate in investigations as appropriate. This requires the RP to keep appropriate logs for ACCESS relevant systems.
Be able to determine if resources are affected by a particular vulnerability, and work with their staff to patch or mitigate.
Protect sensitive information (phone contacts, PGP keys, wiki accounts, etc) as it relates to ACCESS and the AIRTG
Make any local security and privacy policies available and easy to find for ACCESS users who may be running jobs on their systems. For example Acceptable Use, Incident Response, etc.
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: Derek Simmel & Shane Filus, CONECT Cybersecurity Program
Last revised date: 07/13/2023
Retired date:
Data and Network Integration
Infrastructure Integration Roadmap Task
Task Type(s): Technology, Support
Start by phase: Integration
Complete by phase: Operations, Ongoing
RP role(s): Data storage, networking, and sysadmin contact(s)
Summary
Meeting data transfer requirements requires an understanding of storage system, application requirements, and site network connectivity. While available data transfer options are determined by site preferences, ACCESS Networking and Data Transfer Services (NDTS) network engineers and data transfer specialists will continue to facilitate those options by: 1) offering a Globus subscription through at least Project Year 1 (ending 31-Aug-2023); 2) identifying, evaluating, and encouraging the use of promising data transfer applications; 3) providing the CONECTnet Internet2 overlay network to improve connectivity between RPs, and 4) consulting with sites upon request to help them tackle data transfer application and infrastructure challenges.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Understand your requirements
To help in planning for adequate data movement and networking capability, the following aspects of your resource, cyberinfrastructure, and applications should be considered.
Storage and file transfer applications
What part(s) of your resource needs to be accessible for data transfer into and/or out of your site?
The RP should identify what resources it wishes to make available over CONECTnet. Ideally these are just “ACCESS resources”, probably including a perfSONAR network performance measurement server. However, it is not inappropriate to include the entire Science DMZ.
For bulk data transfer (datasets in ~100GB range and larger on ACCESS CONECTnet):
Describe your storage system characteristics, types, and partitions
POSIX filesystems are currently the norm in ACCESS, but some new resources have other storage—tape archives, object stores, cloud storage—so please think about which should/should not be available for ACCESS data transfer. Are any of these parallel/high-performance filesystems, such as HPSS, Lustre, or GPFS?
Do you have a throughput goal for data transfers?
What transfer throughput do you anticipate your ACCESS users will need in order to accomplish their research objectives?
Does your site have a Science DMZ for supporting bulk data transfer?
Other than bulk data transfer, does your resource provide applications that have particular network throughput requirement characteristics, e.g., real-time streaming data, interactive, streaming video, etc?
Do any of these applications require special network features (e.g., QoS, reservations)?
Does your site currently, or are you willing to, host a network performance monitor (perfSONAR) to help ACCESS monitor and diagnose end-to-end network behaviors?
Data and networking survey and consultation
The ACCESS ecosystem offers a Layer3 VPN (L3VPN) provisioned on Internet2 (CONECTnet) to provide connectivity between RPs. Participation in CONECTnet provides valuable performance metrics collection/reporting as well as availability statistics and error reporting. NDTS will be working with Internet2 to identify and expand the available metrics in an effort to increase our visibility and understanding of the types and volumes of traffic traversing CONECTnet.
Please contact the NDTS team t3-ndts@access-ci.org to inform us that you intend to connect to CONECTnet
We will send you a link to a network connectivity survey. Your responses are valuable to help us understand your site’s connectivity and data transfer priorities.
When you complete the survey we will schedule a consultation to review the information and follow up on any questions.
Network Connectivity - WAN connections
To integrate with CONECTnet follow these steps:
The RP should identify a router (RP router) “suitably close” to their ACCESS resources. The definition of “suitably close” is very site specific. Members of the NDTS team are available to help with this determination.
The RP, possibly in conjunction with their campus and/or Internet2 connector (i.e., their regional network provider), needs to configure a VLAN from the RP router to the interface on the router that peers with Internet2. Note the VLAN tag value. Jumbo frames (9000-octet IP MTU) should[1] be supported by the devices in the VLAN.
The RP (or its campus or Internet2 connector) needs to set an ACL in OESS to allow access by the CONECT workgroup.
The RP needs to pick IP addresses for the point-to-point connection between the RP router and the Internet2 router. Either public or private (ULA/RFC 1918) addresses will work, as will any convenient prefix length (for example, PSC and NCSA both use /127 and /31). Clearly, IPv4 needs to be configured; IPv6 is optional but recommended.
The RP now needs to pass along to the CONECT NDTS group the IP address block(s) for the point-to-point connection (indicating which addresses are for the RP end and which are for the Internet2 end), along with the VLAN tag and an autonomous system number (ASN; either public or private) for BGP peering. Optionally, a password for BGP peering can be specified, as can the desire to use BFD (bidirectional forwarding detection).
The CONECT NDTS group will provision the connection on the CONECTnet L3VPN.
The RP can now bring up BGP peering. Note that no prefix filtering is done on the network.
If the RP wishes to configure connections to multiple Internet2 routers, the above steps can be repeated for each connection. The RP is responsible for any traffic engineering (eg, local preference or MED).
The RP…
Network Performance Measurement
(will add text pointers to existing perfSonar documentation)
Data Transfer
The supported file transfer applications for ACCESS are currently scp, sftp, rsync, and Globus. scp, sftp, and rsync are commonly used file transfer applications. Other than verifying that the servers are running (system administrators) and accessible (network engineers for site firewalls; system administrators for IPtables, firewalld, etc.), these applications likely will not require special configuration and management for use by ACCESS participants.
Globus requires specific system, application, and (potentially) hardware configuration. Please see the ACCESS CONECT document Deploy Globus Endpoint for detailed guidance.
Document Management
Status: Official
Official date: 4/24/2023 <mm/dd/yyyy>
Coordinators: Kathy Benninger, ACCESS Operations
Last revised date: 3/11/2023
Retired date:
[1] If jumbo frames are not configured, Path MTU Discovery must be enabled across the end-to-end path.
Deploy Globus Endpoint v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Operations
RP role(s): Data and networking contact(s), System administrator(s)
Summary
A Globus endpoint connects a system to Globus and enables authorized researchers (members of an allocated project team) to transfer data to and from it using Globus. During the last two years of XSEDE, hundreds of researchers used Globus to transfer data to and from XSEDE resources as a part of their projects. A single transfer request may involve a single file or a million files. Individual requests were commonly 100 GB or more and frequently 1 TB or more. Requests of 10 TB or more happened daily. The other end of the data transfer may be a personal system, a campus system (hundreds of campuses have multi-user Globus endpoints and encourage their researchers to use them), or another HPC system (hundreds of national-scale service providers, e.g., DOE or non-US systems, have Globus endpoints). Globus enables researchers to perform transfers at these scales and is broadly available on the systems where research data is used.
Effort
Operators role(s) that would typically perform this task:
RP networking - design network connectivity/topology for data transfer nodes (DTNs)
RP data services - identity storage systems available to DTNs, access policies, and performance requirements
RP operations - deploy DTNs, install and configure Globus Connect Server, evaluate & optimize performance
The following estimates assume starting from scratch. If you have already deployed a Globus endpoint for your resource(s)—especially if the endpoint already uses Globus Connect Server 5.4 software—you will have significantly less work to do. In particular, performance evaluation and optimization should carry over from XSEDE to ACCESS with minimal change. GCS 5.4 endpoints can support both XSEDE and ACCESS using the same DTN and software with additional configuration.
The following assumes you have already acquired DTN servers with appropriate network interface(s) and memory, and network hardware with an appropriate topology for high-performance data transfers. Requirements for these should have been produced by roadmap task 2.1, Data & Networking integration design.
Task |
Estimated effort/time |
|---|---|
Deploy DTN hardware with the required network connectivity |
1 person-day effort (1 week start-to-finish) |
Provision local accounts on DTNs |
2 person-day effort (2 weeks start-to-finish) |
Mount and configure POSIX storage on DTNs |
1 person-day effort (½ week start-to-finish) |
Install and configure Globus Connect Server on DTNs (inc. access policy configuration) |
1 person-day effort (1 week start-to-finish) |
Evaluate and optimize performance |
1 person-week effort (3 weeks start-to-finish) |
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
The following assumes you have already performed roadmap task 2.1, Data & Networking integration design. That task should have produced requirements for DTN hardware (CPU, memory, network interfaces) and network topology (e.g., DTNs hosted in a high-bandwidth zone with connectivity to both the wide-area network and the RP’s data storage system(s)).
This also assumes you have acquired the necessary DTN servers and network hardware. (I.e., this doesn’t discuss acquiring hardware.)
Deploy DTN hardware with the required network connectivity
Data Transfer Nodes (DTNs) are multiuser computer servers that run the data transfer software (Globus Connect Server) that handles the local end of data transfers to or from your data storage systems. Task 2.1, Data and Networking integration design, produced requirements for the RP’s Data Transfer Nodes (DTNs) based on how you anticipate ACCESS researchers will use your resource(s).
Your operations personnel will deploy the DTN hardware (servers/nodes) in your data center.
The network interfaces will be configured per the requirements from task 2.1. (This typically includes one interface to the RP’s high-bandwidth, wide-area network zone and a second interface to the zone where the RP’s data storage is connected.)
This step is complete when your data storage personnel can login to the DTNs and set up storage connectivity, your accounting or operations personnel can login and configure accounts for authorized researchers, and your networking personnel confirm the network interfaces are configured as expected.
Provision local accounts on DTNs
NOTE: We assume that individuals who are part of ACCESS projects that have allocations to use this resource are given local accounts on your resource. If this assumption is not valid for your resource, this section will need to be customized for your resource.
Globus uses local accounts for primary data access.[1] For primary data access to work as expected, individuals who are part of ACCESS projects that have allocations on this resource should have local accounts on the DTNs as well. In later steps, Globus will be configured to use these accounts.
Your accounting or operations personnel will login to each DTN and configure the appropriate mechanism for provisioning local accounts. Each RP has its own account mechanism, so ACCESS cannot provide detailed instructions.
Mount and configure POSIX storage on DTNs
POSIX-accessible data storage must be mounted on the DTNs to be accessible by Globus.[2] Task 2.1, Data and Networking integration design, identified the POSIX storage system(s) on your resource that should be accessible for data transfers.
Your data storage personnel will login to each DTN and configure the appropriate POSIX mounts for POSIX data storage that should be accessible for data transfers per the requirements produced by task 2.1.
Tape archives, HPSS, GPFS, and other mass storage systems may have special tuning parameters and configuration interfaces. If these storage systems are intended to be accessible for data transfers, the requirements produced by task 2.1 should include guidance regarding performance tuning parameters and other settings.
Install/configure Globus Connect Server on DTNs
Globus Connect Server is the software that connects your storage system(s) to Globus so researchers can use Globus to request data transfers to and from your resource. Globus Connect Server supports multi-DTN configurations, both to simplify DTN maintenance (adding/removing DTNs without disrupting transfers) and to enhance performance (Globus will use all available DTNs for transfers). Globus Connect Server can be used with POSIX-mounted data storage or (with optional subscription add-ons) a wide variety of object storage (e.g., OpenStack Ceph, BlackPearl, ActiveScale), cloud storage (AWS, Google, and Microsoft), and specialized research data storage systems (iRODS, Hadoop HDFS).
Globus Connect Server features modular configuration with a variety of access policies, so you can configure a single Globus Connect Server installation to (separately) support a series of distinct communities (ACCESS being one of these communities) that each have distinct IAM and data access policies.
ACCESS provides an IAM policy and support mechanisms that enable RPs to map individual members of ACCESS-allocated projects to local accounts. Task 2.1, Data and Networking integration design, identified your data access policies for data transfers.
Guidance for installing and configuring Globus Connect Server for use with ACCESS is available in the ACCESS Globus Connect Server Installation Guide. The ACCESS guide provides ACCESS-specific guidance, but for general Globus instructions it frequently refers to Globus’s Globus Connect Server v5 Installation Guide, so your personnel will need both resources for complete instructions.
A summary of the steps for installing and configuring Globus Connect Server for use with ACCESS follows.
Your DTN administrator(s) will install the Globus Connect Server packages and ACCESS’s OAuth Mapfile utility package on each DTN using the native package management system.
Your DTN administrator will configure the ACCESS OAuth Mapfile tool to generate and continuously update a table that maps ACCESS identities to the corresponding local accounts.
On one DTN, your DTN administrator will create a storage gateway that limits access to storage systems per your ACCESS data transfer policy, identified in task 2.1.
On one DTN, your DTN administrator will create one or more collections within the ACCESS storage gateway. Each collection provides data access to a specific data storage area per your data access policy.
On any remaining DTNs, your DTN administrator will install a copy of the configuration key file from the first DTN, adding each DTN to the logical Globus endpoint and synchronize configurations across all participating DTNs.
Evaluate and optimize performance
After your Globus endpoint is set up and configured, you should evaluate the performance of the endpoint to confirm that it satisfies the requirements determined in task 2.1. Test endpoints were deployed as part of ESnet’s 2019-2021 Data Mobility Workshop & Exhibition. They are hosted by a variety of well-connected institutions, are pre-loaded with sample reference datasets, and the endpoints and reference datasets are listed on the Globus website.
To evaluate whether your endpoint is performing as expected or not, the simplest method would be to repeat a variety of transfers from the test endpoints to your new endpoint (or vice-versa, with write-enabled test endpoints). Hint: You can use Globus’s Timer feature to schedule a transfer to repeat at preset intervals and collect the results later from the Timers tab on the Activity page. Set multiple timers with different endpoints and reference datasets and compare results.
Document Management
Status: Official
Official date: 4/24/2023 2/15/2023
Coordinators: Lee Liming, ACCESS Operations
Last revised date: 2/3/2023
Retired date:
Incident Response and Coordination v1
Infrastructure Integration Roadmap Task
Task Type(s): Support
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Cybersecurity and incident response contact(s)
Summary
Resource Providers and ACCESS Tracks must follow incident response and coordination procedures as defined in ACCESS’s Incident Response Policy. Representatives from each RP and Track must be identified to participate in the ACCESS Incident Response Trust Group (AIRTG). This representative’s contact information must be provided and will be used in the case of an incident as defined in the Incident Response Policy.
Effort
Identifying representatives for the AIRTG and submitting names: ~1 hour.
AIRTG meetings occur weekly and take less than 30 minutes.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Submit an ACCESS ticket using the Support Portal ticket form with the following information:
Is your issue related to a resource? Not related to a resource
Is your issue related to allocations? No
Please Select an ACCESS category: ACCESS Security
Problem Synopsis: ACCESS AIRTG participants
Problem Description: Technical staff names and email to be added to the AIRTG
You will receive a response from ACCESS CONECT Cybersecurity Group indicating that the individuals have been added to the ACCESS ACCESS Incident Response Trust Group (AIRTG) and the secure communication channels outlined in the effort section above.
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: Derek Simmel & Shane Filus, CONECT Cybersecurity Program
Last revised date: 07/13/2023
Retired date:
Infrastructure Description v2
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Resource or Service Integration Coordinator, Documentation and knowledge base contact(s),
Summary
ACCESS infrastructure that is visible to researchers/users, resource providers, developers, between ACCESS projects, or the broader community must be described. ACCESS resource providers, ACCESS projects, and other ACCESS affiliated online service providers (i.e. science gateways, etc.) enter basic information about their organization and then describe the infrastructure that they operate in the Cyberinfrastructure Description Repository (CiDeR). Audience relevant summaries of these descriptions are displayed in different ACCESS websites.
Initial setup involves the integration coordinator obtaining administrator access to CiDeR so they can authorize others in their organization to enter and maintain infrastructure descriptions. This task should take ~½ hour. Organization staff enter initial descriptions and maintain accurate infrastructure descriptions in CiDeR. This task should take ~1 hour/year per infrastructure element.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in your Integration Roadmap Description.
Detailed Instructions
Organization staff access CiDeR at https://cider.access-ci.org/login.
Initial CiDeR Setup
If your organization already has CiDeR access, skip to step C.
Submit an Integration and Operations Request from this page with the Title “New CiDeR Organization” and the following information:
Organization logo URL or attached file
Organization public URL
Organization PI/director name
Organization PI/director email address
External Data Posting
The ACCESS usernames of individuals that will be CiDeR administrators for your organization. These individuals will be able to grant other individuals access to maintain infrastructure descriptions. Persons can be both CiDeR administrators and the maintainers of descriptions.
Submitted organization information will be entered by a CiDeR administrator in a form as shown below and used to grant access to CiDeR administrators.
Provide CiDeR access to other organization staff
After the integration coordinator has been granted CiDeR access they can grant other organization staff access to enter and maintain infrastructure descriptions:
Select “Service Providers” or “Organizations” along the top.
Click on your “Organization” name.
Click on “Administrators”.
Add additional Organization Admins who may enter and update your organization resource information.
Enter Infrastructure Descriptions
Organization staff enter new infrastructure descriptions as detailed in the Quick Start Guide and User Guide:
https://cider.access-ci.org/cider_manual_full.pdf
Steps:
Select “Resources” along the top.
Click on “Add a Resource” to add a new resource, or click on a resource name to edit an existing resource.
Enter as much information as possible in this first form
Select Resource Features as follows:
Applicable Resource Type |
Features |
|---|---|
Compute, Cloud, Storage |
Select from the following features: |
Compute, Cloud, Storage |
If your resource supports science gateways select features starting with “Science Gateway *”. |
Online Services |
Select from the following features: |
Science Gateway Online Services |
This feature designates a registered Online Service as a science gateway: |
Select a “Resource Type” at the bottom of the screen and “Create Resource”
For Compute, Cloud, and Storage type resources, enter as much information as possible in this second form and the following minimum required information:
Field Name |
Applicable Resource Type |
Notes |
|---|---|---|
Node Count |
Compute, Cloud |
For cloud record the number of controller nodes. |
CPU Core Count Per Node |
Compute, Cloud |
For heterogeneous clusters use the mean value per node |
Memory Per CPU in GB |
Compute, Cloud |
For heterogeneous clusters use the mean value per node |
Local Storage per Node in GB |
Compute |
For heterogeneous clusters use the mean value per node |
Peak Teraflops |
Compute |
Leave blank if unknown |
Disk Size in TB |
Compute, Storage, Cloud |
Total amount of shared storage (network attached and parallel file-system) |
Tips for storage resource providers: The following topics are particularly helpful in your storage resource description.
Intended use - Some storage resources are intended to be used only in conjunction with a specific compute resource, while others are intended to be used without any associated computing. Your intended use statement should clearly state which of these is true for your resource.
Backup policy - If your storage resource isn’t backed up, it’s best to note that in as many places as possible! This is a good place to clearly state the backup policy.
Additional allocation requirement - If your storage resource may only be allocated in conjunction with another resource (compute, cloud), it’s helpful to clearly state that requirement in the resource description.
Variation from standard allocation rates - Most ACCESS storage resources have an exchange rate of one ACCESS allocation credit to one resource unit, and one resource unit to 1 GB of storage. If your resource has a different rate, please state that clearly in the resource description.
Enter Resource Conversion Factors
Skip this section if your resource is not ACCESS allocated.
Page 16 of the CiDeR Manual has instructions for entering a “conversion factor”. This is the NU conversion factor of your resource, divided by 21.576. This will be used to compute your initial marketplace exchange rate. **For most compute resources, you should NOT change the conversion factor that you already have listed.
**In a later stage of ACCESS, additional tools will be created to manage your variable marketplace exchange rate.
Maintain infrastructure descriptions
Information in CiDeR must be kept up to date and reviewed for accuracy at least yearly.
Document Management
Status: Official
Official date: 4/24/2023 2/15/2023
Coordinators: JP Navarro, ACCESS Operations; Nathan Tolbert, ACCESS Allocations
Last revised date: 2/3/2023
Retired date:
Knowledge Base v2
Infrastructure Integration Roadmap Task
Task Type(s): Support
Start by phase: Integration
Complete by phase: Operations
RP role(s): Documentation and knowledge base contact(s)
Summary
The purpose of this task is to ensure that RPs have provided all of the knowledge base elements that are needed to fully represent their resources to the ACCESS team and the general user community. Components of the information provided will be represented on the ACCESS MATCH Portal (AMP).
The following information should be available through CiDeR to ensure that your resource is properly represented on the ACCESS Portal (RAMPS & AMP).
1.1 Summary Knowledge Base Entry for Resource - Confirm summary paragraph describing the Resource Provider offering(s) is available in CiDeR. This will be displayed on AMP but content will be provided via CiDeR API.
1.2 RP User Guide - Full User Guide(s) will reside at and be maintained by the RP on their site or in the knowledgebase (Atlassian Confluence) and accessed by the AMP via the confluence repository.
1.3 Affinity Group - Ensure an Affinity group is created with coordinators and profile is up to date. Submit events and news items via the ACESS Support Portal (ASP) portal.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Creation of an Affinity Group
You will need to log-in first to make changes to your group. Your Affinity group will be automatically created and populated but we request that you ensure you have the proper coordinators and profile that best suits your needs. If you do not see your Affinity Group please submit a request via the link at the top of the page or submit a ticket. An Affinity Group is made for each RP and then additional ones can be requested if you would like to form one that targets a specific set of community members who have a common, shared interest in a computational issue, scientific or engineering research endeavor, diversity and inclusion effort, or any other connection point. Each Affinity group will have access to Slack channels, forums, news, and outage alerts, events and training materials.
News Items
Please submit news items via the ASP Portal. You will see a box on the right side that says “Posting News, Do you have news you would like to share with the ACCESS community?” which will take you to a page that asks for specific details of your news item and the ability to post a news event directly to your/an Affinity group. You will need to be signed-in to ACCESS to submit.
Events
Please submit events via the ASP Portal. This link will take you to a listing of all events in a calendar format. You will see a box on the right side that says “Posting Events, Do you have events or training you would like to share with the ACCESS community?” which will take you to a page that asks for specific details of your event you would like to post. This can also be shared with your Affinity group. You will need to be signed-in to ACCESS to submit.
Document Management
Status: Official
Official date: 4/24/2023 01/27/2023
Coordinators: Alana Romanella, ACCESS Support
Last revised date: 01/27/2023
Retired date:
Local Services ACCESS IAM Integration v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Operations
RP role(s): System administrator(s), Cybersecurity administrator(s)
Summary
This task provides guidance for Resource Providers that want to integrate their services with ACCESS identity management.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Register for an ACCESS User ID
Follow the Register a Web App for ACCESS Auth instructions
Additional information
Related Identity and Access Management (IAM) instructions and documentation:
NetSage Integration
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Planning
Complete by phase: Operations
Operator role(s): Leadership, data and networking contact
Summary
NetSage is a measurement and monitoring tool that collects and evaluates common network measurement data to help users understand patterns of behavior with large-scale data transfers. The plan is for ACCESS Resource Providers to work with NetSage to gather SNMP and Flow data which will be part of the XDMoD system as well as displayed in ACCESS-specific NetSage Dashboards.
For more information see the Using NetSage to understand ACCESS Data Transfer presentation
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Information to deploy a docker container to collect flow data is available online at : https://netsage-project.github.io/netsage-pipeline/docs/deploy/docker_install_simple/
Utilization Reporting for non-ACCESS allocated resources
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Operations
RP role(s): UUtilization data contact(s), System administrator(s)
Summary
ACCESS Metrics provides ACCESS XDMoD and various data reporting and analysis services for ACCESS-allocated resources and for other NSF-funded projects such as the Campus Cyberinfrastructure program.
This task involves setting up the data transfer of utilization data for a compute resource that is not allocated by ACCESS. This includes compute resources that are funded by NSF’s CC* program that are expected to work cooperatively with ACCESS award activities and services.
The data reporting will be via a Federated XDMoD portal that is managed by ACCESS Metrics. This federated XDMoD portal is based on the Open XDMoD software and is configured to be able to report across multiple NSF funded projects and resources. The data will be made available to NSF staff.
Required information:
Resource Specification information including: number of compute nodes, number of compute cores, number of GPU devices, whether resource is node or core allocated, whether compute node sharing is enabled.
Information about the system users: username on compute resource, first name, last name, organization name
Resource manager accounting log data.
Desirable information:
Unique identifiers for system users such as an ORCID iD or ACCESS ID.
Allocation information such as project name, project PI, project field-of-science, funding agency
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
The required log data should be periodically sent to ACCESS Metrics. ACCESS Metrics supports two types of data transfer mechanism. The choice of data transfer mechanism depends on the RP preferences:
Data transfer via REST endpoint (preferred)
OR
Globus Transfer with scheduled transfers
The REST endpoint data transfer mechanism is preferred for small amounts of log data transfer (less than 1 MB per day). the Globus Transfer mechanism is more appropriate for large file transfers (but can be used for smaller amounts of data if desired).
REST data transfer
To send data to ACCESS via REST you need an API Token. We will generate one and
email it to you. Below are examples of how to send the data using curl and
how to send data using a python script.
Example of data transfer using curl command line tool:
curl --form file='@FILE_TO_UPLOAD' -H 'Content-Type: text/plain' -H "Authorization: Bearer YOUR_TOKEN_HERE" https://data.ccr.xdmod.org/logs
where YOUR_TOKEN_HERE should be set to the token that was provided to you by ACCESS Metrics and FILE_TO_UPLOAD should be set
to the filename of the file to upload (note the ‘@’ character before the filename is required).
Example python script to upload the file using the python requests library:
#!/usr/bin/env python3
# Tool for POSTing data from standard input to the UB CCR XDMoD data endpoint.
# Replace 'YOUR_TOKEN_HERE' below with the API token you received from the
# XDMoD team.
import requests
import sys
def main():
if len(sys.argv) < 2:
print("Usage: {} [FILENAME]".format(sys.argv[0]))
return
api_token = 'YOUR_TOKEN_HERE'
file = {'file': open(sys.argv[1], 'rb')}
response = requests.post(
'https://data.ccr.xdmod.org/logs',
files=file,
headers={
'content-type': 'text/plain',
'authorization': 'Bearer ' + api_token,
},
)
print('Response: ', response.status_code, response.text)
if __name__ == '__main__':
main()
Globus Transfer
To setup Globus Transfer, please provide the name or names of the globus accounts that will be used for data transfer. ACCESS Metrics will create a globus share and give write permissions to the user accounts provided by the RP. Instructions for setting up the periodic transfers using the globus tools are either Globus timer cli tool or the web-based Globus Timers.
Required Log files
Log Type |
Description |
Filename |
Update Frequency |
|---|---|---|---|
Resource Manager Log Files |
Daily dumps of the resource manager log files. Use the format described in the Open XDMoD shredder documentation. |
Use the filename direct from the resource manager logs, or if using slurm’s |
Daily |
Name mapping |
Mapping between the username on the resource and a persons full name. Use the format described in the Open XDMoD Names documentation. For Regional Computing resources that support users from multiple different institutions then the User’s instittion should also be included in the file contents as an additional column after the user’s last name. |
|
Upload new copy of the file when new users are added to the resource. |
Resource Specifications |
Information about the number of compute devices (nodes, cores, GPUs) on the resource. The file format supports tracking resource changes over time. Use the format described in the Open XDMoD resource specs |
|
Whenever the resource size changes |
Document Management
Status: Draft
Official date: TBD
Coordinators: Joseph White, ACCESS Metrics
Last revised date: 2023-09-15
Retired date:
Online Service Documentation v1
Infrastructure Integration Roadmap Task
Task Type(s): Support
Start by phase: Integration
Complete by phase: Operations
RP role(s): Service Contact(s)
Summary
The purpose of this task is to ensure that the online service operator provides the documentation needed by service users. Documentation should be published through one of these channels: an access-ci.org website, the Confluence Knowledge Base, or the service’s own web sites.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Using access-ci.org websites
ACCESS projects may document their online services on their own website or the website of other ACCESS projects where appropriate.
ACCESS Confluence
Online services may provide documentation through one of the following ACCESS confluence spaces, where appropriate:
ACCESS Collaboration Portal - for service only used by other ACCESS services
ACCESS Documentation / Knowledge Base - for services used by ACCESS users
See the Knowledge Base integration task
If a separate workspace may be appropriate, contact the ACO to discuss and request it
Example: ACCESS Ticket System Documentation workspace
Online Service Hosted Documentation
Online services with their own web sites may publish their documentation through those sites.
Describing where documentation is
See the Infrastructure Description task for how to record the documentation URL in the CiDeR online service description.
Operational Status Communications v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Resource news and outages publisher(s), System administrator(s), Researcher support contact(s)
Summary
ACCESS resource providers RPs, ACCESS projects (tracks), and online service operators communicate planned and unplanned outages, and configuration changes, using the ACCESS Operations Infrastructure News system.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Initial Setup
First, staff responsible for posting infrastructure news must have logged in once to the ACCESS Operations portal at https://operations.access-ci.org/ before they can be granted access.
Next, the Resource Integration Coordinator must submit the names of staff that need permission to post infrastructure news by opening an Operations Request through the Operations Portal Help Page. Please use the Request Title “ACCESS Operations Infrastructure News Editor access” and the ACCESS Operational Support Issues “Operations: Portal”. Staff that logged in per the previous paragraph will then be granted Infrastructure News Editor permissions.
Recurring Activity - Status Communications
Authorized staff can:
End Date:
For Reconfigurations leave the End Date blank unless it is temporary
For Outages leave the End Date blank if unknown, then update the news item once the End Date is known
Distribution Options:
Select how to communicate the new item. As of February 2023 the only implemented option is “Email only subscribers”. Other Distributions Options are under development.
Update previously posted news:
Posts can be edited to make minor corrections or to update the End Date for outages.
To communicate an significant update to news, edit the posting and prepend the following to the beginning of the original Content:
Update as of <Month-Day-Year>
<update text>
Original News
<original news content>
Posted/updated infrastructure news should appear in the ACCESS Support Portal - Outages Page within 10 minutes.
API Access - Status Communications
Resource providers and ACCESS projects may access infrastructure news through an API at:
Under the “News” section.
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: JP Navarro, ACCESS Operations
Last revised date: 6/8/2023
Retired date:
Performance Data Reporting v1
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Operations
RP role(s): Metrics and performance data contact(s), System administrator(s)
Summary
ACCESS Metrics provides ACCESS XDMoD and various data reporting and analysis services. XDMoD provides reporting on a wide range of CI metrics. This includes CI efficiency reporting that is available to end users, PIs and RP staff.
The main data source for the efficiency reporting is low-level performance data collected from each CI resource. CI resource providers must periodically transfer a copy of the low-level performance data to ACCESS Metrics so that it can be processed and displayed in XDMoD.
This task involves setting up the data transfer of performance data.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
The low-level performance data files must be periodically sent to ACCESS Metrics. The recommended configuration is to use daily increments data transfers. ACCESS Metrics support three types of data transfer design. The choice of data transfer mechanism depends on the RP preferences:
Globus Transfer with scheduled transfers
Data transfer over ssh push from RP to ACCESS Metrics data transfer node (dtn)
Data transfer over ssh pull from RP with login via the ACCESS Metrics Gateway account (Community User xdtas)
Our preferred mechanism is to use Globus Transfer with scheduled and regularly repeating transfers for the data exchange. To setup Globus Transfer, please provide the name or names of the globus accounts that will be used for data transfer. ACCESS Metrics will create a globus share and give write permissions to the user accounts provided by the RP. Instructions for setting up the periodic transfers using the globus tools are either Globus timer cli tool or the web-based Globus Timers.
If option (2) is chosen then the RP should provide an ssh public key and a preferred username that will be used for the data transfers. ACCESS Metrics will provide the name of the data transfer node (dtn). The RP should then use a cronjob (or similar) to transfer the files each day.
Option (3) can only be used if the resource supports ACCESS community accounts and the “Community User xdtas” is authorized. In this case the xdtas account must have read permissions on the low-level performance data files. ACCESS Metrics will manage the periodic transfers.
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: Joseph White, ACCESS Metrics
Last revised date: 2023-02-01
Retired date:
Publish Dynamic Resource Information v2
Infrastructure Integration Roadmap Task
Task Type(s): Technology
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): System administrator(s)
Summary
This task involves installing and running the Information Publishing Framework (IPF) tool on ACCESS allocated compute resources in order to publish batch resource, queue, and job information, software module information, and (optionally) job events to the ACCESS information sharing platform. The ACCESS information sharing platform provides information to ACCESS portals, SGCI resource schema compatible science gateways, and other service developers.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Install, configure, and run IPF per these instructions:
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: Eric Blau & JP Navarro, ACCESS Operations
Last revised date: 7/3/2023
Retired date:
Request RP or Site Staff Allocation v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Any
Complete by phase: Operations
RP role(s): PI and co-PI(s), Resource integration coordinator(s)
Summary
RPs may request a site/staff allocation via ACCESS, to use for supporting and testing the ACCESS allocation workflow.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
To request a site/staff allocation for your site, follow the standard allocations request process (see below), noting in the “public overview” section of your submission that this is for a site/staff allocation.
You will want to start on the ACCESS Allocation website.
Before submitting a request, you will want to start with the overview of the different ACCESS opportunities.
Look across the menu bar in the middle of the page. Click on “Prepare requests”
Within “Prepare requests,” click on “Overview.”
Here you can look at all the different ACCESS opportunities and see which opportunity you would like to submit into.
PLEASE READ THIS PAGE VERY CAREFULLY.
The information on this page will let you know about the credit limits and any additional information about things needed in the submission.
Now that you have decided which opportunity to submit into, you can now work on your submission.
Look across the menu bar in the middle of the page. Click on “Manage allocations”
Within “Manage allocations,” click on “Submit a Request.”
You are now at the Available Opportunities Page.
Look for the Opportunity (Explore, Discover, Accelerate, Maximize) you would like to submit into.
Once you see that Opportunity, click on the button that says “SUBMIT AN (Opportunity) ACCESS REQUEST”
Request Science Gateway Allocation
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Operations
Operator role(s): Gateway administrator
Summary
A science gateway provider needs to have an active allocation for their science gateway. This can be done before or after the Science Gateway Description v1 tasks have been completed.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
ACCESS allocations are exchanged for computing time and other resources or services provided by ACCESS resource providers. Allocations can be used for provisioning computing resources for software executions on high performance computers, for data storage, and for virtual machines to host services.
The science gateway provider requests an ACCESS allocation through the standard process described at https://allocations.access-ci.org/.
Under certain circumstances, a science gateway provider can request multiple allocations:
A gateway provider can have a separate research allocation.
A gateway provider who operates multiple science gateway can have one allocation per gateway.
Effort level varies by the request track. See https://allocations.access-ci.org/ for more information. Science gateway providers can request allocations of any size.
Depending on the size of the allocation request, this step could take days to weeks before the award is made.
Request Science Gateway Community Accounts
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Operations
Operator role(s): PI, CoPI, or gateway allocation administrator
Summary
Science gateways use community accounts to provide single accounts for all gateway users.
Note : One community account must have been attached only to one science gateway.
Prerequisite tasks
The gateway provider must have completed Science Gateways Description
Support Information
For assistance with this task see the Support Information section in the Infrastructure Integration Roadmap Description.
Detailed Instructions
Community accounts are UNIX accounts on ACCESS resource providers that can be used by a science gateway’s users through the science gateway’s user or programming interface. Science gateway providers request community accounts in individual ticket requests through the ACCESS ticketing system directed to the ACCESS Resources Providers, the ACCESS Integration Coordinator assigned can help make these requsts.
Navigate to the ACCESS ticketing system
Select “ACCESS Integration and Operation Support Requests” on “What can we help you with?”
Select “Allocations:XRAS” on “ACCESS Operational Support Issues”
Select “ACCESS Resource Provider Resource” on “Infrastructure Type”
Select the “ACCESS Resource Provider Resource” from the dropdown
Submit
Followup with any discussions on the ticket
Once the ticket is submitted, the allocations team would do what is necessary and keep posting on the ticket. When it’s done, the science gateway provider should confirm on the ticket that the community accounts are created and accessible. It may take hours to days for accounts to be created after registration.
The science gateway provider may need to take additional, resource provider-specific steps to enable the account. This may include, for example, providing specific IP addresses or registering SSH keys that the science gateway will use when accessing the resource provider.
Request Science Gateway Resources
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Operations
Operator role(s): PI, CoPI, or gateway administrator
Summary
Science gateway operators request that their awarded allocation credits be exchanged for computing and storage allocations on specific resource providers resources that advertise as being available to science gateways.
Prerequisite tasks
The science gateway operator must have an awarded allocation. See the document at Request Science Gateway Allocation v1. Depending on the size of the request, this could take days to weeks to complete.
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
The gateway provider uses the ACCESS allocation request system to transfer ACCESS credits to particular resource providers. See https://allocations.access-ci.org/.
The exchange request may take days or longer before approved by the resource provider.
The science gateway provider uses standard ACCESS procedures to convert ACCESS allocation tokens to computing time or storage on target resources. The gateway provider should make sure that the resource provider supports science gateway usage, which should be included in the resource description.
Resource Metrics Data Availability Assessment v1
Infrastructure Integration Roadmap Task
Task Type(s): Integration
Start by phase: Planning
Complete by phase: Integration
RP role(s): Metrics and performance data contact(s), System administrator(s)
Summary
ACCESS Metrics provides ACCESS XDMoD and various data reporting and analysis services. XDMoD provides reporting on a wide range of CI metrics. This includes CI efficiency reporting that is available to end users, PIs and RP staff. The main data source for this is low-level performance data collected from each CI resource.
Resource Providers do not need to install or maintain XDMoD. ACCESS Metrics provides the service that processes the low-level performance data, correlates it with other CI metrics, analyzes it and stores in ACCESS XDMoD. For traditional HPC resources, performance data collection is typically achieved by running monitoring software on the compute nodes. Examples of such software include tacc_stats, Performance Co-Pilot Prometheus or Ganglia to name a few. Performance data can also be obtained from some resource managers (such as slurm) with the appropriate configuration.
The choice of performance data collection system depends on the resource architecture and operational constraints. ACCESS Metrics can provide guidance and recommendations based on our experience.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
If the CI resource is already collecting performance data (or data collection is already part of the design plan) then the details of what data are collected should be provided to the ACCESS Metrics team via the Support Contact.
If the CI resource is not collecting performance data then this should be reported to the ACCESS Metrics team via the Support Contact and we will start the discussion about what are the appropriate performance metrics to collect and the most efficient mechanism used to collect this. For traditional HPC resources, performance data collection is typically achieved by running monitoring software on the compute nodes. Examples of such software include tacc_stats, Performance Co-Pilot Prometheus or Ganglia. ACCESS Metrics can provide guidance and recommendations for the most appropriate tool for the task. ACCESS Metrics team members maintain the tacc_stats software and can provide expert assistance with integration. ACCESS Metrics supports Performance Co-Pilot and Prometheus for Open XDMoD and can also provide assistance with install and configuration.
Document Management
Status: Official
Official date: 4/24/2023
Coordinators: Joseph White, ACCESS Metrics
Last revised date: 2023-02-01
Retired date:
Resource Provider Forum Participation v1
Infrastructure Integration Roadmap Task
Task Type(s): Coordination
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): PI and co-PI(s), Resource integration coordinator(s)
Summary
The Resource Provider Forum (RP Forum) is a key venue for information sharing and coordination among the RP community, and between RPs and the ACCESS program itself. Participating members of the RP Forum will engage in high-level coordination activities and information-sharing meetings with all ACCESS program tracks and other participating RPs. Until the RP Forum is chartered by the ACCESS program, during the first program year, the XSEDE SP Forum will continue to fulfill this role.
Prerequisite tasks
None
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
The RP Forum By Laws are also available for review.
Article 3 is the relevant section for RP Forum application.
These instructions for ACCESS-integrated RPs are also listed in the next section.
Detailed Instructions
The PI, co-PI, or Resource Integration Coordinator submit a request to join the RP Forum to Jeremy Fischer <jeremy@iu.edu> and copy Nicole Wolter <nickel@sdsc.edu> and include:
Relevant NSF grant numbers
The organization name
The resource name
A brief description of the resource and its relevance to the community
The names, email addresses, and roles of the person(s) that will participate in the RP Forum and receive RP Forum emails.
Within 2 weeks you will receive a response from the RP Forum Coordinator, and the designated RP Forum participants will start receiving RP Forum meeting invites and other communications through an RP Forum email list.
Document Management
Status: Official
Official date: 4/24/2023 2023/02/01
Coordinators: Jeremy Fischer, RP Forum Chair & Nicole Wolter, RP Forum Vice-Chair
Last revised date: 2024/03/28
Retired date:
ACCESS Affinity Groups
Version 1 - 2022/09/26
Summary
ACCESS uses affinity groups to enable communications among members of communities. Science gateway providers should join the science gateway affinity group and the affinity groups of any resource providers that the science gateway uses.
Prerequisites
The gateway should have completed registration. See “Science Gateway Registration”.
Effort
See https://support.access-ci.org/affinity_groups for a listing of groups and steps for joining.
Science gateways may want to also create their own affinity group.
Description
Resource providers use affinity groups to communicate with their users, including science gateway providers. The science gateway provider may wish to communicate this information to their users through their own communication channels.
ACCESS science gateway community members and ACCESS staff communicate using the science gateways affinity group. This may for example include changes to policies, new services, and informal support.
Some resource providers may have additional communication channels.
Science Gateway Allocation Application
Version 1 - 2022/09/26
Summary
A science gateway provider needs to have an active allocation for their science gateway.
Prerequisites
None
Effort
Effort level varies by the request track. See https://allocations.access-ci.org/ for more information. Science gateway providers can request allocations of any size.
Depending on the size of the allocation request, this step could take days to weeks before the award is made.
Description
ACCESS allocations are exchanged for computing time and other resources or services provided by ACCESS resource providers. Allocations can be used for provisioning computing resources for software executions on high performance computers, for data storage, and for virtual machines to host services.
The science gateway provider requests an ACCESS allocation through the standard process described at https://allocations.access-ci.org/.
Under certain circumstances, a science gateway provider can request multiple allocations:
A gateway provider can have a separate research allocation.
A gateway provider who operates multiple science gateway can have one allocation per gateway.
Science Gateway Community Accounts
Version 1 - 2022/09/26
Summary
Science gateways use community accounts to provide single accounts for all gateway users.
Prerequisites
The gateway provider must have completed registration. See “Science Gateway Registration.”
Effort
The science gateway provider should confirm that the community accounts are created and accessible. It may take hours to days for accounts to be created after registration.
The science gateway provider may need to take additional, resource provider-specific steps to enable the account. This may include, for example, providing specific IP addresses or registering SSH keys that the science gateway will use when accessing the resource provider.
Description
Community accounts are UNIX accounts on ACCESS resource providers that can be used by a science gateway’s users through the science gateway’s user or programming interface. Science gateway providers request community accounts as part of the registration process. Accounts are created by ACCESS internal mechanisms.
Science Gateway Description
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Integration
Operator role(s): Gateway administrator
Summary
The science gateway provider registers their gateway with ACCESS to provide publicly accessible and internally accessible information about the science gateway.
Prerequisite tasks
The science gateway provider must have completed Science Gateway Resource Requests
Support Information
For assistance with this task see the Support Information section in the Infrastructure Integration Roadmap Description.
Detailed Instructions
The science gateway provider must create an Integration and Operations Request
Set ACCESS Operational Support Issues to ACCESS-wide: Provider Integration - Infrastructure Integration and Roadmaps
Set Infrastructure Information: Science Gateway and specify the Science gateway name
Include all the details of the gateway in the summary. Please use the following template.
Institution Name:
Science Gateway Name:
Acronym:
Public URL:
Short Name:
Short Description:
Associated allocation ID(s):
Status (in development or in production):
Production date (when did/will it become):
Gateway PI:
First name, Last name <email>
Gateway co-PIs:
First name, Last name <email>
First name, Last name <email>
Gateway administrator(s):
First name, Last name <email>
First name, Last name <email>
Cybersecurity and incident response contact(s):
First name, Last name <email>
First name, Last name <email>
Integration coordinator:
First name, Last name <email>
Integration coordinator is the contact person from the science gateway side working with the ACCESS concierge on the integration process.
Once the ticket is created, it will be reviewed by the ACCESS concierge and will get back to you for additional information. And then the ACCESS concierge would create/update the information in the ACCESS website. The task is completed when the gateway information is published to the ACCESS website.
ACCESS may periodically review registration information to confirm that it is correct. Gateway providers should create subsequent tickets in the future to update the information.
Science Gateway Registration
Version 1 - 2022/09/26
Summary
The science gateway provider registers their gateway with ACCESS to provide publicly accessible and internally accessible information about the science gateway. This step also triggers the creation of community accounts on allocated resources.
Prerequisites
The science gateway provider must have completed “Science Gateway Resource Requests”
Effort
The science gateway provider must provide information in web forms (see description) that should take approximately an hour to complete.
The task is completed when the gateway information is published to the ACCESS web site and when community accounts are created on allocated resources.
ACCESS may periodically review registration information to confirm that it is correct. Gateway providers should promptly update their registration information.
Description
Science gateway providers register their gateway with ACCESS, providing the following information:
The gateway’s status, which can be “in production” and ready to accept users, or “in development”.
The URL for the science gateway (displayed by ACCESS if the gateway is also in production).
The fields of science supported by the science gateway (displayed by ACCESS if the gateway is in production).
A short description of the gateway that can be displayed by ACCESS if the gateway is in production and which can help potential users find the gateway.
Contact information for the gateway provider and operators, which can be used by ACCESS and resource providers to contact the gateway operators in case of technical problems, cybersecurity issues, improper use of systems, etc.
A short name for the science gateway that complies with UNIX account name requirements, which may be used as community account names by Resource Providers.
Allocations Required Data Fields
Assuming we’re going to target a “Discover ACCESS” project as the default.
Required Fields
Project Title
Abstract (overview)
Keywords (note: this could just be entered as “gateway”, so not really needed on the registration form)
Primary Field of Science
Personnel (at least a PI). A technical or alternate contact would be useful.
These individuals would need to have ACCESS IDs.
Supporting Grant (if one exists)
Funding agency
Grant title
PI Name
Start/end dates
Grant number
Field of Science (probably same as Primary FOS above)
Awarded Amount
Program Officer name & email
Officially, Discover ACCESS requires a 1-page “proposal”, but we could probably just allow them to attach a PDF of their gateway registration.
Resources needed
They’ll get ACCESS Credits first, but —if they know—it would be useful to capture what resources they intend to use, and we can make the first credit Exchange for them. We may need to define a default “gateway package” or amount to start with. If they don’t know the resources they intend to use, that step can be deferred until they do know.
Science Gateway Resource Requests
Version 1 - 2022/09/26
Summary
Science gateway operators use their allocations to request computing and storage from ACCESS resource providers that support science gateways.
Prerequisites
The science gateway operator must have an awarded allocation. See “Science Gateway Allocation Application”. Depending on the size of the request, this could take days to weeks to complete.
Effort
The gateway provider uses the ACCESS allocation request system to transfer ACCESS credits to particular resource providers. See https://allocations.access-ci.org/.
The exchange request may take days or longer before approved by the resource provider.
Description
The science gateway provider uses standard ACCESS procedures to convert ACCESS allocation tokens to computing time or storage on target resources. The gateway provider should make sure that the resource provider supports science gateway usage, which should be included in the resource description.
Science Gateway Usage Reporting
Infrastructure Integration Roadmap Task
Task Type(s): Coordination, Support
Start by phase: Integration
Complete by phase: Operation
Operator role(s): Gateway administrator
Summary
Science gateways that execute jobs on ACCESS resource provider resources report usage through an ACCESS-provided REST service. Usage statistics are available through XDMoD.
Prerequisite tasks
The gateway must have an allocation and a community account. See Request Science Gateway Community Accounts v1.
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Science gateways that submit jobs to high performance computing and other resources must provide additional information, such as the user name, to supplement usage information associated with community accounts that is gathered automatically by the XDMoD system. This allows resource providers and access to more accurately track and report the number of users of their systems.
ACCESS provides a simple REST API for this reporting. Science gateways can also use other API methods to programmatically access usage information for individual users of the community account.
The science gateway provider should follow the instructions at https://allocations-api.access-ci.org/acdb/.
After integration, the gateway provider should verify that usage is being correctly reported in the ACCESS XDMoD system, https://xdmod.access-ci.org/. The gateway provider can also use REST API calls to verify information reporting; see https://allocations-api.access-ci.org/acdb/.
Service Metrics Data Availability Assessment v1
Infrastructure Integration Roadmap Task
Task Type(s): {Coordination, Technology, Support}
Start by phase: {Planning, Integration, Operations}
Complete by phase: {Planning, Integration, Operations, Ongoing}
RP role(s): Metrics and performance data contact(s), System administrator(s)
Summary
<Summarizes the purpose or goal for the task, who performs the task, and approximate effort. If this task replaces or upgrades one or more other tasks, provide references to those tasks and summarize significant new or changed elements.>
Prerequisite tasks
None
OR
<task name and link>
…
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
<REPLACE the above boilerplate sentence if you want a different operator support process for this task>
Detailed Instructions
<Detailed task instructions that may include separate sections for both one-time / setup tasks and ongoing or recurring activities.>
Document Management
Status: {Draft, Official, Retired}
Official date: 4/24/2023 <mm/dd/yyyy>
Coordinators: <name>, <ACCESS project>
Last revised date: <mm/dd/yyyy>
Retired date: <mm/dd/yyyy> or blank
Ticket Handling v1
Infrastructure Integration Roadmap Task
Task Type(s): Support
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Researcher support contact(s), most other RP contacts
Summary
**NOTE** This task will be updated incrementally as the transition from Request Tracker (RT) to Jira Service Management (JSM) is completed.
ACCESS resource and online service operators will be assigned tickets for issues or questions about their resources and online services. They will monitor the ticket system for tickets assigned to them, triage them as necessary, reassign the request to other staff or organizations if necessary, address requests they are able to address, and close the request once it is resolved.
Prerequisite tasks
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
Detailed Instructions
Ticket Routing Queue Setup
If your organization/RP integrated other resources or services in the past and already has ticket routing queues, you can use those for new resources and you will not need to request new ones.
To request new organization/RP specific routing queues submitting an ACCESS ticket using the form listed here https://access-ci.atlassian.net/servicedesk/customer/portal/2/group/3/create/32.
Please use the “ACCESS Operational Support Issues” of “ACCESS-wide:Ticket system - ACCESS related ticketing system”. In response to your request an ACCESS ticket system administrator will configure the ticket system with queues for your organization/RP and inform you when they are configured. Please review the “For RP queues” and “For ACCESS awardee queues” sub-sections below for additional details on what to include in your ticket.
For RP queues
If you are a resource provider (RP) integrating a resource, provide a short organization name or abbreviation that can be used to define your RP queue. Theis queues will be named “<short_name>”, like “Jetstream-2” or “Delta”.
Identify for this queue the name and ACCESS usernames of:
People that can UPDATE tickets in the queue
Person who will be assigned tickets by default (this person will be able to reassign tickets to anyone else with queue access)
For ACCESS awardee queues
If you are an ACCESS awardee integrating a service, provide a short name or abbreviation for new track specific queues you want setup. The queue names should follow these guidelines:
Identify for each new queue the name and ACCESS usernames of:
People that can UPDATE tickets in the queue
Person who will be assigned tickets by default (this person will be able to reassign tickets to anyone else with queue access)
Also provide any keywords related to your services that would help individuals recognize that a ticket should be routed to this queue. For example, an ACCESS-ACO-Support queue might have keywords “confluence” or “access wiki” associated with it to indicate that access wiki or confluence issues should be assigned to this queue.
Ticket Handling
Resource and online service operating organization staff will receive email from the ticket system, or access the ticket system online at:
Tickets will be assigned to a queue and agents can assign tickets to themselves from their queue.
Ticket handling typically involves these activities.
Reassign Externally, Accept, or Reassign Internally
The default assignee or anyone with update access to tickets in the queue should first determine if the ticket was properly assigned and if not, reassign to a different queue.
If the ticket was assigned to the correct queue, they may reassign it to anyone else in their organization with ticket system access, or retain ownership of the ticket.
FOr more details please see the documementation https://access-ci.atlassian.net/wiki/spaces/ATSupport/overview
Document Management
Status: Draft
Official date: 2/15/2023
Coordinators: JP Navarro, ACCESS Operations; Alana Romanella, ACCESS Support
Last revised date: 2/3/2023
Retired date:
Ticket Handling v2
Infrastructure Integration Roadmap Task
Task Type(s): Support
Start by phase: Integration
Complete by phase: Ongoing
RP role(s): Researcher support contact(s), most other RP contacts
Summary
ACCESS resource and online service operators will be assigned tickets for issues or questions about their resources and online services. In response they will monitor the ticket system for tickets assigned to them, triage them as necessary, reassign them to other staff or organizations if necessary, resolve issues, and close tickets once the request is addressed.
Prerequisite tasks
Support Information
For assistance with this task or with using the ticket system:
Open an ACCESS Integration and Operation Support Request using using this page
In the new ticket form select the “ACCESS Operational Support Issues” type “ACCESS-wide: Ticket System - ACCESS-related Ticketing Systems“
Detailed Instructions
Ticket Routing Queue Setup
If your organization/RP integrated other resources or services in the past and already has ticket routing queues, you can use those for new resources and you will not need to request new ones.
To request new organization/RP specific routing queues open an ACCESS Integration and Operation Support Request as described above. A ticket system administrator will configure the ticket system with queues for your organization/RP and inform you when they are configured. Please review the “For RP queues” and “For ACCESS awardee queues” sub-sections below for additional details on what to include in your ticket.
For RP queues
If you are a resource provider (RP) integrating a resource, provide a short organization name or abbreviation that can be used to define your RP queue. Theis queues will be named “<short_name>”, like “Jetstream-2” or “Delta”.
Identify for this queue the name and ACCESS usernames of:
People that can UPDATE tickets in the queue
Person who will be assigned tickets by default (this person will be able to reassign tickets to anyone else with queue access)
For ACCESS awardee queues
If you are an ACCESS awardee integrating a service, provide a short name or abbreviation for new track specific queues you want setup. The queue names should follow these guidelines:
Identify for each new queue the name and ACCESS usernames of:
People that can UPDATE tickets in the queue
Person who will be assigned tickets by default (this person will be able to reassign tickets to anyone else with queue access)
Also provide any keywords related to your services that would help individuals recognize that a ticket should be routed to this queue. For example, an ACCESS-ACO-Support queue might have keywords “confluence” or “access wiki” associated with it to indicate that access wiki or confluence issues should be assigned to this queue.
Ticket Handling
Resource and online service operating organization staff will receive email from the ticket system, or access the ticket system online at:
Tickets will be assigned to a queue and agents can assign tickets to themselves from their queue.
Ticket handling typically involves these activities.
Reassign Externally, Accept, or Reassign Internally
The default assignee or anyone with update access to tickets in the queue should first determine if the ticket was properly assigned and if not, reassign to a different queue.
If the ticket was assigned to the correct queue, they may reassign it to anyone else in their organization with ticket system access, or retain ownership of the ticket.
Fr more details please see the documementation https://access-ci.atlassian.net/wiki/spaces/ATSupport/overview
<Template Task Title> v<n>
Infrastructure Integration Roadmap Task
Task Type(s): {Coordination, Technology, Support}
Start by phase: {Planning, Integration, Operations}
Complete by phase: {Planning, Integration, Operations, Ongoing}
RP role(s): {list RP role(s) involved in this task}
Summary
<Summarizes the purpose or goal for the task, who performs the task, and approximate effort. If this task replaces or upgrades one or more other tasks, provide references to those tasks and summarize significant new or changed elements.>
Prerequisite tasks
None
OR
<task name and link>
…
Support Information
For assistance with this task see the Support Information section in the Integration Roadmap Description.
<REPLACE the above boilerplate sentence if you want a different operator support process for this task>
Detailed Instructions
<Detailed task instructions that may include separate sections for both one-time / setup tasks and ongoing or recurring activities. Explain the roles that will perform this task an approximate effort required.>
Document Management
Status: {Draft, Official, Retired}
Official date: <mm/dd/yyyy>
Coordinators: <name>, <ACCESS project>
Last revised date: <mm/dd/yyyy>
Retired date: <mm/dd/yyyy> or blank