Incident Reporting Procedures

To keep its user base as informed as possible, OIT makes public any interruption, change, or degradation of any service that it provides. SysNews (http://sysnews.ncsu.edu) is the core method of communicating these deviations of service. ALL outages and service degradations, planned or unplanned, will be reported via SysNews unless other methods have been pre-established with the recipients of those services.  Services often have other methods of communication (e.g., online_notify, MyPackPortal) which may be used in addition to SysNews.

Types of posts

Planned incident

Any change (or potential change) in service should be posted, even if it has no direct effect on service usability. SysNews can show future events, allowing people to inquire about and plan for them. Posting future events that have no predicted user impact also informs users in case something goes wrong and usability is inadvertently impacted.  Timing of advanced notification is dependent upon the event, but should be communicated as far in advance as practical.

Unplanned incident

A SysNews post should be made immediately upon the discovery of any unplanned outage or incident, regardless of the immediately available information. “Service X is acting sluggishly, and OIT staff is aware and working on a solution” allows users to know that a problem exists, and it saves the Help Desk from fielding a lot of calls from users reporting the same incident.

Follow-up posts

  • Update a SysNews post regularly until the issue is resolved, even if no new information is available.
    The definition of “regular” depends on the service and the impact, but hourly is a good baseline.
  • Include the expected frequency of updates if known.
  • Do not include an estimate on service recovery unless it is relatively certain.
  • Staff members are encouraged to use the NC State Help Desk for updating, as it both frees technical staff from updates and makes the Help Desk aware of the current situation.
  • Submit a final post when the service has returned to normal status.

How to make a post

  1. Go to http://sysnews.ncsu.edu.
  2. Log in to SysNews via the login link under “User Info”
  3. Authorized posters will see a “Post a News Entry” link under “System Announcements”.
    If this link is not visible, please request access via the Help Desk at help@ncsu.edu.   The Help Desk is available to assist with posting when needed.
  4. To create a new post, click the “New Post” button.
    To create a follow-up post, do one of the following:

    • Select the original post and click on “Followup On Post.”
    • Use the “Post a Followup On This Entry” link at the top of the original post.
  5. Choose the “Organization” that is most directly responsible for the affected service, using the generic “OIT” organization only if the service is cross-functional or the incident is more broad-reaching. Most posters will only have rights to post events for their own organization. The NC State Help Desk staff has posting rights in multiple organizations.
  6. Choose a “Category” and “Type” based on the incident itself. Note that Type is optional.
  7. Create a “Subject” for your post that will be informative to the general public at a glance. “uni00mr memory leak” means nothing to most folks, whereas “Email service issues” at least gives a general idea about what is wrong.
  8. The “Reply To” setting for SysNews is usually set to help@ncsu.edu, unless the need for direct contact elsewhere is required.
  9. If this is a planned incident, put in the “Date of Future Event” and “Time of Future Event”. This will list the incident in the “Upcoming System Events” section of SysNews, and will bump the original post to the top of SysNews on the day of the event. Note that “Time” is not required at all, and only a begin time can be used if necessary.
  10. In the “Public Message” section, briefly summarize the problem in layman’s terms. Focus on the problem itself, the services (and parts of a service) that are affected, and the possible impact of the problem. Keep technical details to a minimum.
  11. In the “Technical Details” section you can give specific details about an issue. This usually includes patch versions, server hostnames, and other things that the general public doesn’t care about but technical staff might need to know. Note that Technical Details is restricted only by Unity ID – anyone with an active Unity ID can read the technical details – so do not include sensitive information.
  12. SysNews is a website, but it also has a shadow email list that gets copies of each post as it is made. If this is a follow-up, the option to “Repost to list” is available. Otherwise, the post automatically goes out to the mailing list for all new posts.
  13. SysNews can cross-post to many “Optional Mailing Lists” if the need arises. This is limited to external email addresses (as opposed to email client distribution lists). Additions to the list can be requested through the Help Desk.
  14. You can opt to exclude the technical details from the cross-posted lists, but this is generally discouraged.

Who can post?

Anyone authorized through the SysNews tool to make posts can make posts during an incident.  Individual groups may establish their own internal processes for making SysNews posts. Often it is very practical for individual OIT employees to make posts regarding their own services.

Help

  • If needed, the NC State Help Desk can help you make a SysNews post during normal business hours.
  • The Help Desk has authorization through SysNews to update posts for other employees, should the need arise.
  • An after-hours Help Desk pager rotation (reachable via pager.ncsu.edu or through Operations) is available in the event of a catastrophic system outage.

 

Last updated 06/05/08