Companies big and small are constantly trying to get control of their IT departments so they run more efficiently, but often struggle to implement good working practices. As we’ve worked with several IT departments to implement processes for reduced downtime, we’ve learned some key things that positively impact corporations. See how we helped one client transform their IT program from lackluster to flourishing in just a short time.
The back story:
A few years ago, we brought on a new client in the hospitality space. At the time, they had a few configurations that were less than ideal, for their users and for us as their administrators. The existing network and Active Directory setup were an amalgamation of years of unmanaged change, reactive break/fix support, and lack of IT vision and strategy.
The issues:
1. Disjointedness
Our client has five unconnected locations across Madison, from an IT perspective this introduced complexity and overhead because each location was operating as its own entity with separate Active Directory domains and isolated file shares hosted on a single physical server at each location.
2. Collaboration challenges
The staff at each location used local server storage to maintain event data, financials, scheduling spreadsheets in addition to location-specific operations items. Since there was no easy way to share this information across locations, each staff purchased third party solutions at additional costs and resorted to sharing login information.
3. Backups (or lack of)
We identified early on that no off-site or off-server backups were happening. Because each location kept a unique set of files, we were forced to create and manage separate backup jobs for each site. This added more overhead and complexity for IT.
4. User and Policy Management Challenges
With regular staff turnover, we found dozens of accounts that hadn't been used in months or even years, and we discovered that users had accounts at multiple sites that were not synchronized. Login scripts, unmanaged drive mappings, printer management, and NTFS permissions were piecemealed, making general administration a real challenge. User accounts were also poorly named making it difficult to correlate to a real person.
5. Antiquated, inconsistent gear
Much of the networking, workstation, and server gear was outdated and not necessarily suited for the changes we were implementing. Sites were using dual homed servers (a dual homed server is one that maintains network identity in two disparate networks) as firewall/routing devices, and network switches were found in the strangest places (behind storage boxes, under desks, in the ceiling).
6. Unsafe networking practices
With WIFI available to customers, guest users were added to the same default network as the workstation and server devices. In addition, IoT devices (jukeboxes, dartboards etc.) were also put into the same default network, leaving their systems vulnerable to attack unnecessarily. This configuration was also leading to DHCP pool exhaustion because of the high turnover of devices. This became evident on several occasions when users were suddenly unable to connect to the network because they couldn't be assigned an IP address.
The solution
Before we could get into really improving the dis-connectivity and collaboration of the business, we took care of a few basics.
1. Inventory
We inventoried, tagged, and recorded every single piece of IT equipment in every location and prioritized immediate replacement of at-risk devices.
The act of tagging equipment was immediately valuable in our ability to provide remote support. Instead of staff describing which piece of equipment wasn’t working properly, they could easily identify it using the asset number we assigned and then we could reference our asset management system to understand which device we were dealing with.
2. Introduce process
As part of this inventorying, we also implemented a change management system so that moving forward we had historical records of device changes and we put a stop to ad hoc changes that ended up interrupting services.
We also on-boarded our client to our support process, which includes our ticketing system. All incoming requests are funneled through a single system, giving our teams more control for triage, resolution and communications related to an open issue.
3. Button up
Once we got a handle on the physical networking in the building, we removed the unnecessary external access and replaced the "firewalls" with quality, stateful firewalls that were capable of higher throughput and more robust networking configurations. We removed or replaced some of the unmanaged switching gear, with new managed switches so we could start doing network segmentation.
The final product
Our final plan included creating a new Active Directory domain and moving each of the sites into the new common domain. Distributed File Services Replication (DFSR) will maintain and replicate a common file share between all five locations. All network traffic will be segmented according to business class using VLAN’s and the software defined networking capabilities of the Unifi access points we've installed. By doing this, we’ve eliminated single points of directory failure at each location and can easily maintain backups of company files several times over. Plus, users can login from any site using the same credentials and have access to the same set of files.
From an administrative standpoint, we’ve simplified user and policy management and improved consistency across the sites as well as set the foundation for an easy cloud integration into Office 365.
Because we couldn’t take any single site out of commission for too long, we launched a virtual server in Azure on which to create the new domain. A full mesh of secure tunnels was established between each site and the Azure virtual network enabling fully encrypted communications. From here, we were able to visit each site and migrate them into the shared domain and move files into the shared system.
At each site, the network segmentation and identification was made to be consistent. This allowed authenticated WiFi users to roam between locations without having to re-enter SSID details. We even extended this to the guest network so visitors to one location would have a seamless network experience at another location.
The whole project resulted in a much more cohesive, collaborative, and secure infrastructure using nothing more than the equipment already on hand. It just goes to show that you don’t always need to invest a ton (especially in hardware) to greatly improve your IT infrastructure, just think about using what you have, differently.