Senior Systems Engineer

City & County of San Francisco - San Francisco, CA4.1

Full-time | Temporary$59 - $74 an hour

Applicants are encouraged to apply immediately as this recruitment may close at any time but not earlier than March 21, 2019. The position is open until filled.

The Department of Technology is the centralized technology services provider within San Francisco City & County government, delivering technology infrastructure and services to approximately 33,000 employees and 800,000 residents. The department has an annual operating budget of over $130M and contains over 220 employees. Core service areas include: Infrastructure and Operations, Technology Security, Service Delivery & Management, Enterprise Applications, Project Management Office, Public Safety Systems & Wiring, Technology Administration, Policy & Governance and IT Public Communications.

The primary responsibility of the Senior Systems Engineer is to effectively manage & administer a complex multi-tenant private cloud infrastructure. Aspects of this position include backup, recovery, capacity and disaster recovery planning, provisioning, installation/configuration, operations and maintenance of systems hardware and software, report creation and distribution, service level support.

The emphasis for the Senior Systems Engineer position will manage the architecture, administration, maintenance, development, and implementation of policies and procedures for providing recovery of data and systems.

In this position the incumbent must have the ability to determine, plan & understand the complexities of VDCs, the systems or networking components & requirements to plan for or enhance the needs of the Enterprise system or network. Troubleshoot and resolve VDC problems with minimal disruption to the system or network; exercise judgment, decisiveness and creativity in designing, planning, troubleshooting and integration of large complex systems or networks based on the potential benefits and/or consequences; develop alternative solutions within established guidelines and standards; communicate effectively orally and in writing; establish good working relationships with other IS professionals, vendors, management and departments; read and understand professional journals, manuals and literature; and prioritize competing requests for service.

The 1043 Senior Systems Engineer assignments are in a highly technical area which supports VDCs residing in geo diverse data centers which host multiple City client departments. The Senior Systems Engineer will provide functional and technical expertise; develop and/or modify procedures to solve VDC issues; perform multiple software functions and support hardware platforms.

Essential Duties:
Design, plan, install, test, integrate, implement, document, maintain and enhance the physical and logical components of a large complex system of virtual servers, virtual server platforms and storage as well as 3rd party cloud offerings (Azure, AWS, Google) & Disaster Recovery Architecture. This may involve the analysis and evaluation of systems software, hardware and communications strategies.
Manages disaster recovery and resumption plans.
Collaborate with other City departments in datacenter disaster recovery plan, including remote hot site replication of critical infrastructure
Experience with best practices, and procedures within the VM platforms, VDC Operations, & disaster recovery
Plan, implement and provide senior level support pertaining for Cloud technologies related to PaaS, Saas & IaaS.
Providing Linux OS and Windows Server administration skills to maintain, manage, administer access, control and perform upgrades of the server infrastructure supporting all business and infrastructure host systems on various virtual environment, and traditional implementations
Performing system modifications to accommodate hardware and OS version upgrades
Installing and hardening Linux & Windows operating systems and planning for applying security and recommended patches on a regularly scheduled basis
Performing system modifications to accommodate hardware and OS version upgrades
Managing storage allocation and back-up/recovery operations
Troubleshooting of system performance in collaboration with application support, database administration and network administration groups.
Working and troubleshooting network infrastructure with the Department of Technology staff including switches, routers and firewalls
Working in close collaboration with City Wide technology staff in the support and maintenance of the network, security and hardware support teams to accomplish desired results
Participating in on-call rotation and maintenance of server support and performing other duties as assigned
Ability to thrive in time-sensitive, results oriented environment
Relies on experience and judgment to plan and accomplish goals.
Leads and directs the work of others.
A wide degree of creativity and latitude is required.
Manages daily activities for ensuring the security and integrity of company data, databases, information systems, and technology.
Cross-train in various support areas in order to provide assistance during afterhours on-call rotation
Assist in developing best practices and procedures for Oracle Engineered systems operations
Help support Level 1, 2, 3, troubleshooting with 24x7 support
Document procedures and troubleshooting steps related to all aspects of our hybrid cloud infrastructure, including but not exclusive to underlying hardware comprising compute (blades), storage (SAN/NAS), storage and data networks, and software, Azure cloud; develop and review documentation prior to general distribution.
Assist in establishing policies, procedures and standards. Develop alternative solutions to problems within established guidelines and standards.
Provision or install all underlying virtual platform components, including but not exclusive to compute, storage, and network platforms.
Ensure system recoverability and performance stability of all aspects of software and hardware comprising our hybrid cloud environment encompassing on premise virtual platform and virtual machines running on the platform, both management and tenant.
Perform departmental and system-wide virtual and storage system assessments, including "physical to virtual" and "virtual to virtual" server migrations over dissimilar platforms and technologies. Optimize software and hardware parameters for smooth integration/migration and assist in determining network efficiencies.
Act as the liaison between vendors, technical support hot-lines and departments to resolve complex system problems as needed. Assist in the coordination and implementation of corrective measures; this may involve site visits, telephone assistance, remote systems management, and participation in technical committees. - Participate and/or drive the establishment of standard engineering practices for service, project delivery and documentation.
Respond to service request and incident assignments, validating incident and request information, executing and closing assigned tickets, engaging Operations and Support organizations as needed to provide a positive support experience for the client or tenant.
Perform other related duties as requested.

Job Type:
The Temporary Exempt- Full Time position is excluded by the Charter from the competitive civil service examination process and shall serve at the discretion of the appointing officer. The anticipated duration of this project position is twenty-four (24) months and will not result in an eligible list or permanent civil service hiring.

Nature of Work:
Incumbent must be willing to work a 40-hour per week schedule and may be required to work additional hours when necessary as determined by the department.

Work Location:
Incumbent will conduct the majority of work at the Department of Technology, 1 South Van Ness. However, there may be situations where the incumbent will be required to work at other sites throughout the City of San Francisco as necessary.

Minimum Qualifications
These minimum qualifications establish the education, training, experience, special skills and/or license(s) which are required for employment in the classification. Please note, additional qualifications (i.e., special conditions) may apply to a particular position and will be stated on the exam/job announcement.

An associate degree in computer science or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units [i.e., at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in computer science or a closely-related field].

Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.

Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/ forty-five (45) quarter units with a minimum of 10 semester / 15 quarter units in computer science or a closely related field.

Desirable Qualifications:
Previous as an IS Engineer Senior or related experience in a large enterprise environment that demonstrates the knowledge and ability to perform essential duties; experience working with Linux and Windows Server administration skills to maintain, manage, administer access, control and perform upgrades of the infrastructure supporting all business and infrastructure host systems on private cloud virtual environment, experience working as a VDC support engineer, Solid understanding of load balancing, horizontal scaling, UNIX-related network services, TCP/IP networking, DNS and HTTP(s); Experience in project management; Familiarity with Windows Server OS, WSH, Perl, Visual Basic or batch file commands; Experience with Visio; Ability to interact effectively with team members and all levels of users; and to balance customer needs while adhering to corporate and department policies; Attention to detail and commitment to quality; Ability to develop and follow written procedures and guidelines; Project planning and execution; and Ability to work well with limited supervision; and to multi-task and deal with changing and competing priorities.

Disaster Recovery/Business Continuity Program Development expertise
Experience with disaster recovery policy and management, and process development and improvement.
Senior engineer level understanding of Windows Unix/Linux Operating Systems: Red Hat, Solaris, CentOS
Virtualization: VMware
Storage: Network Appliance, EMC, Oracle Pillar
Strong SAN, NAS knowledge
Experience with virtual/Cloud data center consolidations, relocations, or moves.
Experience with public cloud workload migrations
Experience with DR/BC strategy
Experience with DR Plans/Playbook/Orchestration
Experience with DR Testing and Exercise
Experience in government and/or public safety environment.
Experience working with and configuring VMware vCloud Networking and Security (VCNS), VMware vShield and associated components and appliances.
Hands on experience implementing and supporting VMware vCloud Director 5.x, or vCloud Automation Center 6.x.
Experience implementing and regularly utilizing VMware Operations Center 5.8/6.
Hands on experience supporting other Virtual Machine Technologies such as Microsoft Hyper-V or AWS.
Experience with vCenter Orchestrator (VCO) or Puppet orchestration tools.
In depth knowledge of Cisco Unified Computing System (UCS) Blade Servers and their respective management software platforms.
n depth knowledge of Storage Area Network management tools/software, such as EMC's Unified Infrastructure Manager or NetApp's On Command system, and associated replication methodologies.
Experience with implementing disaster recovery and high availability architectures utilizing SAN and VMware based replication technologies and tools such as VMware Site Recovery Manager (SRM).
Experience with Windows Server 2008/2012, Active Directory, and Red Hat Linux provisioning in a multi-domain environment.
Current Microsoft Certified Systems Engineer certification (MCSE) on Security and Windows Server 2012 Active Directory.
VMware Certified Systems Engineer certifications such as VMware VCAP-DCA, VCAP-CIA and VCP-NV Certification or higher.

Special Requirements:
Criminal Justice Information Services (CJIS) Security Clearance may be required. (See Security Clearances and Background Investigations below).

1. Medical Testing: Prior to appointment, eligible candidates must successfully pass the TB testing process.

2. Security Clearances & Background Investigations: Positions in this classification may require that successful candidates who become eligible for appointment may be required to go through a background investigation to determine the candidate’s suitability for employment in this classification. Factors considered in the investigation may include employment history, use of illegal/controlled substances. Reasons for rejection based on this investigation may include, but not limited to: applicable convictions, repeated or serious violations of the law, inability to accept supervision, inability to follow rules and regulations, falsification of application materials and/or other relevant factors. Failure to obtain and maintain security clearance may be basis for termination.

How To Apply
Applications for City and County of San Francisco jobs are only accepted through an online process. Visit to register an account (if you have not already done so) and begin the application process.

§ Select the desired job announcement

§ Select “Apply” and read and acknowledge the information

§ Select either “I am a New User” if you have not previously registered, or “I have Registered Previously”

§ Follow instructions on the screen

Computers are available for the public (from 8:00 a.m. to 5:00 p.m. Monday through Friday) to file online applications in the lobby of the Dept. of Human Resources at 1 South Van Ness Avenue, 4th Floor, San Francisco.

Applicants may be contacted by email about this announcement and, therefore, it is their responsibility to ensure that their registered email address is accurate and kept up-to-date. Also, applicants must ensure that email from CCSF is not blocked on their computer by a spam filter. To prevent blocking, applicants should set up their email to accept CCSF mail from the following addresses (,,,,,,,,,,

Applicants will receive a confirmation email that their online application has been received in response to every announcement for which they file. Applicants should retain this confirmation email for their records. Failure to receive this email means that the online application was not submitted or received.

All work experience, education, training and other information substantiating how you meet the minimum qualifications must be included on your application by the filing deadline. Information submitted after the filing deadline will not be considered in determining whether you meet the minimum qualifications. Resumes may be attached to the application; however, resumes will not be accepted in lieu of a completed City and County of San Francisco application.

Applications completed improperly may be cause for ineligibility, disqualification or may lead to lower scores.

Note: Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.

If you have any questions regarding this recruitment or application process, please contact the exam analyst, Carol Wong, by telephone at 652-628-5045, or by email at