├── 2. Solution ├── img │ ├── CI_CDPipeline.png │ ├── ServiceMapping.png │ ├── ServiceInteractions.png │ ├── AsIsSysOpSquadAppArch.png │ ├── ToBeSysOpSquadAppArch.png │ └── SysOpsSquad-DR-Deployment.png ├── Readme.md ├── Assumptions.md ├── DisasterRecovery.md ├── CICDRouteToLive.md ├── MinimumViableProduct.md ├── MicroServices.md ├── SolutionOverview.md ├── RisksAndSensitivePoints.md └── MigrationApproach.md ├── 3. Perspectives ├── TicketCreation.png ├── TicketExecution.png ├── TicketToExpertMatching.png └── Readme.md ├── 1. Problem Statement ├── Readme.md ├── Constraints.md ├── Non-Functional.md ├── BusinessDrivers.md ├── FunctionalRequirements.md ├── Stakeholders.md └── BusinessGoalAndScope.md ├── 4. ADRs ├── 001 We are using ADR (template).md ├── 003a Manager will handle all the corner scenarios ADR.md ├── 004 Single User Customer UI segregated by Role.md ├── 005 Use the current Mobile App for the Experts.md ├── 003 Will Use Uber Model ADR.md ├── 006 Migration Strategy Selection.md └── 002 Microservices Approach.md └── README.md /2. Solution/img/CI_CDPipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/CI_CDPipeline.png -------------------------------------------------------------------------------- /2. Solution/img/ServiceMapping.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/ServiceMapping.png -------------------------------------------------------------------------------- /3. Perspectives/TicketCreation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/3. Perspectives/TicketCreation.png -------------------------------------------------------------------------------- /3. Perspectives/TicketExecution.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/3. Perspectives/TicketExecution.png -------------------------------------------------------------------------------- /2. Solution/img/ServiceInteractions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/ServiceInteractions.png -------------------------------------------------------------------------------- /2. Solution/img/AsIsSysOpSquadAppArch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/AsIsSysOpSquadAppArch.png -------------------------------------------------------------------------------- /2. Solution/img/ToBeSysOpSquadAppArch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/ToBeSysOpSquadAppArch.png -------------------------------------------------------------------------------- /3. Perspectives/TicketToExpertMatching.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/3. Perspectives/TicketToExpertMatching.png -------------------------------------------------------------------------------- /2. Solution/img/SysOpsSquad-DR-Deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/archkata2021t17/sysops-squad/HEAD/2. Solution/img/SysOpsSquad-DR-Deployment.png -------------------------------------------------------------------------------- /2. Solution/Readme.md: -------------------------------------------------------------------------------- 1 | # Solution Background 2 | 3 | 1. [Solution Overview](SolutionOverview.md) 4 | 1. [MicroServices](MicroServices.md) 5 | 1. [Continuous Integration and Delivery](CICDRouteToLive.md) 6 | 1. [Migration Approach](MigrationApproach.md) 7 | 1. [Minimum Viable Product](MinimumViableProduct.md) 8 | 1. [DR Deployment](DisasterRecovery.md) 9 | 1. [Risks and Sensitive points](RisksAndSensitivePoints.md) 10 | 1. [Assumptions](Assumptions.md) 11 | -------------------------------------------------------------------------------- /2. Solution/Assumptions.md: -------------------------------------------------------------------------------- 1 | ### Assumptions 2 | 3 | 1. The User Acceptance Testing is already in place for the existing system and will be used for Migrated System 4 | 1. Except for Ticket Assignment scenarios 5 | 1. Expert purchase of spare parts and related activity is as is business - no change required 6 | 1. We are assuming that the current system is low volume system which do not require Elasticity to be implemented 7 | 1. The original development team will work on decomposing the monolith. 8 | 1. The complexity of the system is low-medium. 9 | 10 | -------------------------------------------------------------------------------- /1. Problem Statement/Readme.md: -------------------------------------------------------------------------------- 1 | # Problem Background 2 | 3 | 1. [Business Goals and Context](BusinessGoalAndScope.md) 4 | 1. [Business Drivers and SARs](BusinessDrivers.md) 5 | 1. [Functional requirements overview and pre-analysis](FunctionalRequirements.md): Functional requirements as per the business drivers. 6 | 1. [Non Functional requirements](Non-Functional.md): Non-functional requirements as per the business drivers. 7 | 1. [Stakeholder](Stakeholders.md): Enlist major stakeholders and their general concerns related to overall system. 8 | 1. [Constraints](Constraints.md): Business and technical constraints that impacts architecture design 9 | -------------------------------------------------------------------------------- /2. Solution/DisasterRecovery.md: -------------------------------------------------------------------------------- 1 | # Disaster Recovery Deployment 2 | 3 | As shown in the diagram below, the services will be deployed in an Active-Passive failover configuration, front-ended by a load balancer cluster. The load balancer would allow us to switch from Active zone to the Passive zone, in case all the services in the Active zone are unavailable. Within each zone, multiple instances of each microservice will be deployed to provide the required high availability at individual service level. There would be continuous replication of the database in the Active configuration, to the database in the Passive configuration. 4 | 5 | ![SysOps Squad High Availability Deployment](img/SysOpsSquad-DR-Deployment.png) 6 | -------------------------------------------------------------------------------- /4. ADRs/001 We are using ADR (template).md: -------------------------------------------------------------------------------- 1 | # Record architecture decisions 2 | 3 | Date: 2021-04-29 4 | 5 | ## Status 6 | 7 | Accepted 8 | 9 | ## Context 10 | 11 | We need to record the architectural decisions made on this project. 12 | 13 | ## Decision 14 | 15 | We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions). 16 | 17 | ## Consequences 18 | 19 | See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools). 20 | 21 | **Positive:** If any 22 | 23 | **Negative:** If any 24 | 25 | **Risks:** If any 26 | 27 | **Bonus Features:** If any 28 | -------------------------------------------------------------------------------- /1. Problem Statement/Constraints.md: -------------------------------------------------------------------------------- 1 | # Constraints 2 | 3 | Below listed global constraints for the solution that drives all trade-off decisions: 4 | 5 | 1. Since it could impact business, the migration with all issues resolved, needs to be done as soon as possible. 6 | 2. The cost of making the changes for migration needs to be reasonable. 7 | 3. Development team is limited, so the solution should be simple at the beginning to prove business ideas 8 | 4. Time-to-market is critical, and with support of the previous constraint, it should be super simple to deploy 9 | 5. AWS is the primary target platform for deployment 10 | 6. Budget is limited, so OSS and tools provided by the target platform should end up higher in prioritization 11 | -------------------------------------------------------------------------------- /1. Problem Statement/Non-Functional.md: -------------------------------------------------------------------------------- 1 | # Non functional Requirements 2 | 3 | From the Business Drivers we have the following non-functional requirements: 4 | 5 | | # | Non-functional Requirements | 6 | |----|----| 7 | | 1. | Downtime during migration should complete in after hours. The SysOps Squad application should be up before the next day business hours. 8 | | 2. | BD.004 - The system must be highly available to Customers (internal/external) 9 | | 3. | BD.005 - Resolve the frequent system crash 10 | | 4. | BD.006 - The system must be responsive during the peak business hours 11 | | 5. | The new system should be cheap and fast to develop and deploy: the business is having trouble right now so *cost* and *time-to-market* is essential 12 | -------------------------------------------------------------------------------- /2. Solution/CICDRouteToLive.md: -------------------------------------------------------------------------------- 1 | # Continuous Integration and Delivery 2 | 3 | ### Goals 4 | The route to live should be 5 | 1. Fast 6 | 1. Protect current functionality 7 | 1. Automated 8 | 9 | ### Overview 10 | 11 | The above goals will be realized by the CI/CD pipeline, which includes different test suites to protect current functionality. 12 | 13 | ![CICD](./img/CI_CDPipeline.png) 14 | 15 | The above figure depicts the CI/CD pipeline flow 16 | 1. Developer will check-in the code(including unit test) into version control system. 17 | 1. The CI/CD automation tool (Jenkins) will listen to the check-ins, and on each trigger it will run clean, build, unit tests. 18 | 1. To keep the code quality to an acceptable standard. The static code analysis tool will be used. 19 | 1. After the code analysis is successful, the deployment artifacts will be stored in artifact repository. 20 | 1. The first deployment will be done on Development environment. 21 | 1. If deployment is successful, component and smoke tests will be performed. 22 | 1. Depending on the requirement to deploy on higher environments, authorized user can further push the artifacts, which 23 | will have additional test suites to be tested based on the environment. This cycle can go up to Production. 24 | 25 | 26 | -------------------------------------------------------------------------------- /4. ADRs/003a Manager will handle all the corner scenarios ADR.md: -------------------------------------------------------------------------------- 1 | # 3a. Manager will handle all the corner scenarios 2 | 3 | Date: 2021-04-29 4 | 5 | ## Status 6 | 7 | Approved 8 | 9 | ## Context 10 | 11 | This decision is an extension of decision [001 Will Use Uber Model] 12 | This decision talks about the Tickets which doesn't cover by any Expert (e.g. may be location is quite far away like suburbs etc.) 13 | 14 | ## Alternatives 15 | 16 | #### Auto assignment by Sysops Squad System 17 | 18 | If the customer ticket is not self-assigned by any of the Expert, for instance some remote area is not covered by any 19 | expert. System will wait for a configured time and then further assign it to relevant Expert. 20 | 21 | #### Manually Assign by the Manager 22 | 23 | Since this situation is a clear corner case of loosing ticket - this needs to be handled by Manager. 24 | 25 | ## Decision 26 | 27 | Clearly, this is solving the current problem where tickets are getting lost - we decided that Manager will take additional 28 | responsibility to manage these corner cases (left after implementation of Uber self assignment model). 29 | 30 | ## Consequences 31 | Manager may need to set a side a time to handle these activities along with current activities assigned to him. 32 | -------------------------------------------------------------------------------- /2. Solution/MinimumViableProduct.md: -------------------------------------------------------------------------------- 1 | # Minimum Viable Product 2 | 3 | If the client's situation doesn't permit to full-blown upgrade (from the cost or development time perspective), 4 | we can go with a Minimum Viable Product (MVP). 5 | 6 | Here is how it will be if the client is willing to take this approach. 7 | 8 | 1. The Ticketing flow is fully migrated to the new architecture - it's essential 9 | 2. The Knowledge Base is scraped from the Monolith 10 | * It is just text and images 11 | * It's detached from everything else 12 | * Solution: The fastest and cheapest is to migrate it (as a database) to a stand-alone Wiki server and left 13 | as is for Experts to use/search/update. It's deactivated in the current Monolith. 14 | 4. The Reporting Component stays 15 | * it's only used by the managers and has no requirement for high availability 16 | * it only needs the data from the respective DBs 17 | * Solution: there will be written a script that will extract the necessary data from the microservices databases, 18 | and put in the Monolith DB, after that the Reporting module can be fired. 19 | 5. The Payment Component stays 20 | * Only information needed for the Ticketing flow is if a Customer is covered for this specific date. 21 | * Solution: 22 | * The Customer UI for payment still feeds the data into the monolith 23 | * there will be developed a simple way (like a DB trigger) to feed the "customer paid until this date" 24 | to the new architecture 25 | -------------------------------------------------------------------------------- /4. ADRs/004 Single User Customer UI segregated by Role.md: -------------------------------------------------------------------------------- 1 | # 4. Multiple UI Interface for each Sysops User Role 2 | 3 | Date: 2021-04-29 4 | 5 | ## Status 6 | 7 | Approved 8 | 9 | ## Context 10 | 11 | Decision about the User/Customer UI design. There are 4 different Actors who work for Sysops Squad and 1 other Actor is Customer. 12 | 13 | ### Sysops Users 14 | 15 | ## Expert 16 | ## Admin 17 | ## Manager 18 | ## Customer Service 19 | ## Customer 20 | 21 | 22 | ## Alternatives 23 | 24 | #### Single User/Customer UI segregated by Role 25 | 26 | This is more favorable since this model allows re-use of components and easy to maintenance; however may not be good from 27 | security point of view (people trying to access the system beyond their Role). 28 | Also, when development team needs to update the UI then it needed to regress other's as well. 29 | 30 | #### Individual UIs for each 31 | 32 | This solution was becoming more complex therefore hard to maintain; however this will provide a solution which is currently 33 | a problem where sometime system is not available for anyone even the internal staff. 34 | 35 | ## Decision 36 | 37 | We will go for Individual UIs for each SysOps User Role, the common UI libraries will be used, so that common features 38 | like Authentication and Authorisation can be managed at single place. 39 | 40 | ## Consequences 41 | 42 | Dev team need to maintain multiple UI code but will provide the decoupling which is badly needed to solve the current 43 | problem of System not available ALL users. 44 | -------------------------------------------------------------------------------- /4. ADRs/005 Use the current Mobile App for the Experts.md: -------------------------------------------------------------------------------- 1 | # 5. Use the current Mobile App for the Experts 2 | 3 | Date: 2021-04-29 4 | 5 | ## Status 6 | 7 | Approved 8 | 9 | ## Context 10 | 11 | Decision for re-using the current Mobile app with update of self assigning the tickets. 12 | 13 | ## Alternatives 14 | 15 | #### Use the current Mobile App for the Experts 16 | 17 | ##### Positive points 18 | 19 | With ease of updates to the Mobile application, we can continue using the Mobile app. Experts are already familiar with 20 | Mobile App. 21 | 22 | New update will give them confidence and self management. 23 | 24 | It will help in Ticket Notification 25 | 26 | ##### Negative points 27 | 28 | Unclear how easy to decouple the current Mobile App from the monolith and point to the new architecture. If APIs are not 29 | well thought and segregated, if the code of Mobile App depends on specific replies from the monolith 30 | (coupled to its inner working), then it might be not worth the effort. 31 | 32 | #### Use the Omni channel browser based UI for Experts 33 | 34 | We have the devs who will work on Web UIs for all other roles, they can take over the Expert Mobile App functionality 35 | to the web, and to some extent replicate the behavior. 36 | 37 | The firm is not focusing on best mobile experience for Experts, so it's OK to give them a completely new web-based 38 | experience (which can be even better than the old one) 39 | 40 | ## Decision 41 | 42 | We agreed to go ahead with Use of the current Mobile App for the Expert with reasonable modifications. 43 | 44 | ## Consequences 45 | 46 | Already described above as positives/negative points. 47 | -------------------------------------------------------------------------------- /4. ADRs/003 Will Use Uber Model ADR.md: -------------------------------------------------------------------------------- 1 | # 1. Will Use Uber Model for (Self) Assigment of Tickets 2 | 3 | Date: 4 | 5 | 28th April, 2021 6 | 7 | ## Status 8 | 9 | Approved 10 | 11 | ## Context 12 | 13 | The core issue described in the SysOps squad is as follows 14 | 15 | - never showing up due to lost tickets 16 | 17 | - often times the wrong consultant shows up to fix something they know nothing about 18 | 19 | 20 | ## Decision 21 | 22 | We will be using Uber model for self-assignment of System Ticket. 23 | 24 | ## Consequences 25 | 26 | Experts will take the responsibility of the ticket - will reduce the chances of loosing it down to none 27 | 28 | Correct expertise will take the job instead wrong assignment of job due to data error 29 | 30 | Workers (Experts) can work on their own pace 31 | 32 | 33 | # Positive 34 | 35 | Additional Benefit to employees (Experts) 36 | 37 | Reduce the complexity of managing Ticket Routing and Assignment 38 | 39 | No lost of Tickets anymore 40 | 41 | Right Consultant would choose the job according to their skill-set and location 42 | 43 | # Negative 44 | 45 | There might be some cases where non of the Experts wants to take the ticket - may be a remote location of the city or suburbs. 46 | 47 | # Risks 48 | 49 | If case of tickets are not self-assigned by the Experts 50 | - then Manager would have additional role to play where he will assign the ticket to suitable Expert 51 | 52 | # Bonus Features 53 | 54 | Experts will be compensated additionally if they clear more tickets in a day than expected 55 | 56 | NOTE: The business model for rewards based on visits done in a day is yet to be purposed to business. 57 | -------------------------------------------------------------------------------- /1. Problem Statement/BusinessDrivers.md: -------------------------------------------------------------------------------- 1 | # Business Drivers 2 | 3 | ## Business goal: 4 | 5 | Business goal: 6 | 7 | To improve the current Sysops squad by process re-engineering or redesign to make the system consistence, reliable and customer centric 8 | 9 | ## Business Drivers (BD) 10 | 11 | Loosing business, regular complaints from customers, not sustainable due to losses 12 | 13 | | # | Business Driver | 14 | |----|----| 15 | | BD.001 | Changes to the system must be quick, easy and non disruptive 16 | | BD.002 | Zero Customer's Ticket loss 17 | | BD.003 | 100% correct assignment of tickets to relevant Experts 18 | | BD.004 | The system must be highly available to Customers (internal/external) 19 | | BD.005 | Resolve the frequent system crash 20 | | BD.006 | The system must be responsive during the peak business hours 21 | 22 | 23 | ## Significant Architectural Requirements (SAR) 24 | 25 | List of architecture driving requirements (primary functional, quality attribute, and life-cycle requirements) 26 | 27 | | # | Significant Architectural Requirements | From BD | 28 | |----|----|----| 29 | | SAR.001 | Migrate Monolith application to microservice architecture | BD.001 | 30 | | SAR.002 | Create a CI/CD pipeline for automatic build, test, deploy in all environments | BD.001 | 31 | | SAR.003 | Add Automated Regression Test Suite as part of CI/CD | BD.001 | 32 | | SAR.004 | Re-design the Ticketing system module | BD.002, BD.003 | 33 | | SAR.005 | Deploy the system in Highly available mode | BD.004, BD.005, BD.006 | 34 | -------------------------------------------------------------------------------- /2. Solution/MicroServices.md: -------------------------------------------------------------------------------- 1 | ## Microservices Details: 2 | 3 | The monolith will be divided into several microservices, their details and interaction is given below. 4 | 5 | The microserivce will be interacting on the PUSH Model except the "Reporting Service" which is designed on PULL model. 6 | 7 | ![ServiceInteration](./img/ServiceInteractions.png) 8 | 9 | NOTE: This interaction is based on limited knowledge and understand of the current problem there may be more iteration needed for finalise the interactions. 10 | 11 | ### High level responsibility of each microservice 12 | 13 | #### Login Service 14 | Internal User, Customer Login and Security Logic 15 | 16 | #### Customer Service 17 | Maintain Customer Profile, customer registration 18 | Support contracts for a Customer and Products in the plan 19 | 20 | #### Expert Service 21 | Maintain Expert Profile (Name, Location, skills etc.) 22 | 23 | #### Ticketing Service 24 | Ticket creation, maintenance, completion, common code 25 | 26 | #### Notification Service 27 | Sends the ticket to the experts mobile device app 28 | Sends survey email to customers 29 | 30 | #### KB Service (Knowledge Base) 31 | Maintain and view items in the knowledge base 32 | Query engine for searching the knowledge base 33 | 34 | #### Reporting Service 35 | All reporting (experts, tickets, financial) 36 | 37 | #### Survey Service 38 | Maintain Survey, Capture and Record Survey Results 39 | 40 | #### Expert Shortlisting Service 41 | Find the expert and assign the ticket 42 | Notify Customer that expert is on the way 43 | 44 | #### Billing Service 45 | Customer monthly billing and customer credit card info 46 | Payment History and prior billing statements 47 | 48 | #### User Service 49 | Maintain internal users and role 50 | -------------------------------------------------------------------------------- /4. ADRs/006 Migration Strategy Selection.md: -------------------------------------------------------------------------------- 1 | ADRs - Big bang vs Phased Migration 2 | 3 | 4 | Date: 2021-05-02 5 | 6 | ## Status 7 | 8 | Approved 9 | 10 | ## Context 11 | SysOps squad is an existing application with a number of issues. Also any change / fix in the current application code takes too long and breaks something else. To get rid of these issues, we have proposed a new architecture. Now we need to decide if we should change few components at a time or all components at once. 12 | 13 | 14 | ## Alternatives 15 | 16 | #### Phased Migration 17 | In general Phased migration of one or few components at a time seems like a safe approach. However here it could become error prone since there has been a history of things getting broken with every change. Also it would result in code duplication, since a special arrangement has to be worked out for the migrated components every release. 18 | 19 | #### Big bang Migration 20 | Big bang migration, i.e. migration of the entire application to the new architecture with an automated test regression suite that covers all the functionality, seems less complex and will get us sooner with fewer issues. Many of the existing components will be refactored to work as a separate service with its own database. To further mitigate the risk, we could run the old and new applications in parallel, with both receiving the requests and run tests that compare the results between the old and new applications. 21 | 22 | ## Decision 23 | Big bang migration 24 | 25 | 26 | ## Consequences 27 | 28 | The decision to go for Big bang migration should allow us to get to the new architecture sooner. The automation regression suite is a critical component that should make it possible, and hence needs appropriate focus and investment. Also, a test suite to compare results of new application with that of the old application, needs to be built. 29 | -------------------------------------------------------------------------------- /4. ADRs/002 Microservices Approach.md: -------------------------------------------------------------------------------- 1 | # Microservices approach 2 | 3 | Date: 2021-04-29 4 | 5 | ## Status 6 | 7 | Accepted 8 | 9 | ## Context 10 | 11 | We need to decide what the dominating system approach will be. There are several options like: monolith, microservices, 12 | micro-kernels, modularized monolith. 13 | 14 | ### Alternatives 15 | 16 | Key map: 17 | - + Promotes 18 | - ++ Strongly promotes 19 | - O Neutral 20 | - - Negative 21 | - -- Strongly negative 22 | 23 | | | Monolith | Microservices | Modularized Monolith | 24 | |----|----|----|-----| 25 | | Ease of Deployment | ++ | - | ++ | 26 | | Availability | - | ++ | - | 27 | | Autonomy | -- | ++ | + | 28 | | Traceability | ++ | - | O | 29 | | Performance | ++ | ++ | ++ | 30 | | Modifiability | O | ++ | + | 31 | | Maintainability | - | ++ | + | 32 | | Integrity | ++ | -- | + | 33 | | Security | - | ++ | ++ | 34 | | Scalability | -- | ++ | + | 35 | 36 | ## Decision 37 | 38 | Based on alternatives in the context of the business needs, and a development team we think that microservices is 39 | the best option for now. This will definitely cause some disturbance to business during migration and new process design 40 | but will pay off in longer term. Further, customer would not need to invest again in Tech stack since we are confident that 41 | new architecture model will resolve the problems. 42 | 43 | ## Consequences 44 | 45 | The main benefits come in the development and deployment area. The team could save a lot or resources setting up 46 | dev/test/qa/prod environments. The cost of development and cognitive complexity is low level because most of the code base 47 | has been reused - with minimal changes to the existing sytem. 48 | 49 | We believe that with minimal changes to the functional components and some changes to business process will resolve current issues. 50 | -------------------------------------------------------------------------------- /2. Solution/SolutionOverview.md: -------------------------------------------------------------------------------- 1 | # Solution Overview 2 | 3 | ## Principles 4 | 1. Adding value to the customer. 5 | 1. Self service / Automation where possible. 6 | 1. Use Open source component, reuse before a buy or a build. 7 | 1. Technology Independent. Avoid a vendor lock-in. 8 | 1. Easy to use and maintain. 9 | 10 | ## Summary 11 | After analysing the problem statement and current application architecture of Penultimate Electronics, its identified that 12 | the current system is a monolith, and some key functionalities do not work reliably. 13 | 14 | ##### As-Is Architecture 15 | ![AsIsAppArchitecture](./img/AsIsSysOpSquadAppArch.png) 16 | 17 | ##### To-Be Architecture 18 | The Purposed Architecture is to break the current monolith into microservices architecture. 19 | 20 | **Key Architectural Points** 21 | 1. Current application will be divided into multiple [microservices](./MicroServices.md). 22 | 1. Reuse the services where possible. The existing component to microservice mapping can be found [here](img/ServiceMapping.png) 23 | 1. Ticket assignment service is re-designed as Expert Selection microservice based on UBER model, details can be found [here](../3.%20Perspectives) 24 | 1. Message Broker queues and guaranteed delivery will be used for inter-service communication. 25 | 1. Migration would be done via a series of MVPs which will gradually migrate the entire application to the new microservice-based architecture. 26 | 1. Create robust regression test suite to have confidence in moved, changed and newly created services. 27 | 1. On demand scaling of microservices to achieve high availability. 28 | 1. Disaster recovery of the system will be handled by Active-Passive deployment. Details are [here](DisasterRecovery.md). 29 | 1. Purposed business process changes 30 | 1. Purposed responsibilities addition to the manager role for some corner case scenario's 31 | 1. Expert will commit to the customer visit himself, rather than assigned. 32 | 33 | ![ToBeArchitecture](./img/ToBeSysOpSquadAppArch.png) 34 | 35 | 36 | -------------------------------------------------------------------------------- /1. Problem Statement/FunctionalRequirements.md: -------------------------------------------------------------------------------- 1 | # Functional Requirements 2 | 3 | The business goal is to improve the customer experience of the SysOps trouble ticketing system. Therefore, the functional requirements stay identical to that of the original system. 4 | 5 | We have deduced the following functional requirements from the available documentation: 6 | 7 | | # | Functional Requirements | 8 | |-----|-------------------------| 9 | |1.| A customer should be able to register with their contact details, support contracts and billing information. 10 | || 11 | |2.| A registered customer should be able to raise tickets via web or a call, for any problems with purchased appliances. 12 | || 13 | |3.| Based on the ticket details, an expert with the relevant skills, service area, current location and available at the specified time, should visit the location to fix the problem. 14 | || 15 | |4.| Once an expert is assigned to a ticket, the customer should be notified that the expert is on their way. 16 | || 17 | |5.| An expert should be provided with a knowledge base, to find out what things have been done in the past to fix the problem. 18 | || 19 | |6.| Once the expert fixes the problem, they mark the ticket as “complete” and add information about the problem and repair information to the knowledge base. 20 | || 21 | |7.| After the ticket is marked as "complete", the customer should receive a mail with a link to a survey regarding ticket resolution. The completed survey is recorded. 22 | || 23 | |8.| A customer should be able to check the current status of the ticket via web or phone call. 24 | || 25 | |9.| A customers should be automatically billed monthly based on credit card information contained in their profile. 26 | || 27 | |10.| A customer can view billing history and statements through the system. 28 | || 29 | |11.| An administrator should be able to add / maintain the experts to the system, with the experts' locale, availability, and skills. 30 | || 31 | |12.| A manager should be able to request and receive various operational and analytical reports, including financial reports, expert performance reports, and ticketing reports. 32 | || 33 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Architectural Katas April - May 2021 - Team Arch Mahal (Team 17) 2 | 3 | ## About the team 4 | We, the Arch Mahal team are a group of people, with a varied architectural experience, across a wide variety of domains - finance, 5 | high performance trading systems, telecom and IT infrastructure. Though we have registered as Team 17, we prefer the name Arch Mahal 6 | because of two reasons: 7 | 1. It represents the initials of the team members. 8 | 2. The word "Mahal" means palace in many Indian languages, and palaces are associated with great architectures. 9 | 10 | Since the team members are spread across the time zones - Singapore, India and UK, coordination was a challenge, but it 11 | seems to have worked out since all the team members are passionate about this project and software architecture in general. 12 | 13 | ## Architectural Katas April - May 2021 14 | 15 | ### Problem Definition 16 | In this architectural kata, the teams have been given a project to migrate an existing trouble ticketing application - SysOps Squad to a new architecture. The existing application has several issues with regards to core functionality and availability. This application is a large monolith, where any changes / fixes take too long and usually give rise to new issues. 17 | 18 | ### Resources 19 | - [Architectural Katas, April – May 2021](https://learning.oreilly.com/videos/architectural-katas-april/0636920557906) 20 | - [Miro](https://miro.com/app/board/) for diagrams 21 | 22 | ### Solution Structure 23 | #### Table of contents 24 | - [Problem Statement](1.%20Problem%20Statement/Readme.md) 25 | - [Business Goals and Context](1.%20Problem%20Statement/BusinessGoalAndScope.md) 26 | - [Business Drivers](1.%20Problem%20Statement/BusinessDrivers.md) 27 | - [Functional Requirements](1.%20Problem%20Statement/FunctionalRequirements.md) 28 | - [Non Functional Requirements](1.%20Problem%20Statement/Non-Functional.md) 29 | - [Stakeholders](1.%20Problem%20Statement/Stakeholders.md) 30 | - [Constraints](1.%20Problem%20Statement/Constraints.md) 31 | - [Solution Overview](2.%20Solution/Readme.md) 32 | - [Solution Overview](2.%20Solution/SolutionOverview.md) 33 | - [Microservices](2.%20Solution/MicroServices.md) 34 | - [Migration Approach](2.%20Solution/MigrationApproach.md) 35 | - [CI/CD](2.%20Solution/CICDRouteToLive.md) 36 | - [DR Deployment](2.%20Solution/DisasterRecovery.md) 37 | - [Risks](2.%20Solution/RisksAndSensitivePoints.md) 38 | - [Assumptions](2.%20Solution/Assumptions.md) 39 | - [Perspectives](3.%20Perspectives) 40 | - [ADRs](4.%20ADRs) 41 | -------------------------------------------------------------------------------- /1. Problem Statement/Stakeholders.md: -------------------------------------------------------------------------------- 1 | # Stakeholders 2 | 3 | These are list of all stakeholders for the SysOps Squad 4 | 5 | | Stakeholder | Concerns | 6 | |-------------|----------| 7 | |Customer |entering registration details in the trouble ticketing app should be hassle free.| 8 | ||system should always show the registration details and current support contract accurately| 9 | ||entering problem details in the trouble ticketing app should be hassle free.| 10 | ||once the ticket is created, it should remain in the system unless explicitly closed.| 11 | ||notification regarding the expert's allotment and when they will arrive needs to reach customer without fail.| 12 | ||system should allow for negotiation of time slot if the expert is not available at the specified time.| 13 | ||expert should arrive in the agreed time slot, and resolve the problem.| 14 | ||ticket should get resolved as quickly as possible and at minimum cost.| 15 | ||if closure of the ticket is going to take longer, customer should get the current status and possibly an ETA for when the problem will actually get fixed.| 16 | ||once a ticket is closed, it should not be required to reopen a resolved ticket or cause new problems with the device.| 17 | ||charges should be according to the support plan.| 18 | ||system should show accurate billing history and statements.| 19 | ||| 20 | |Expert |ticket needs to be assigned to an expert based on their skills.| 21 | ||ticket needs to be assigned to an expert based on their current location and the time it takes to travel from the current location to the new location.| 22 | ||ticket needs to be assigned to an expert based on their available time slot| 23 | ||expert should get the notification of a new ticket as soon as possible.| 24 | ||on resolution of problem, expert should be able to mark the ticket as resolved / complete. The state should reflect accurately in the system database.| 25 | ||on resolution of problem, system should make it very easy for the customer to rate the expert.| 26 | ||knowledge base application is available to check for existing details, gives the existing information and also allows entering new details easily.| 27 | ||system database needs to have the right and up-to-date details about expert's skills, locale, availability, past resolved tickets and customer reviews.| 28 | ||| 29 | |Administrator |entering details of experts should be easy and should remain unchanged in the system unless explicitly modified. 30 | ||management of billing and processing for the customer should not result in accounting inaccuracies. 31 | ||static reference data should remain in the system unless explicitly modified. 32 | ||| 33 | |Manager | in day-to-day operations, system shows accurate information about all the office staff - experts and administrators.| 34 | ||for escalated tickets, all the right details are present to take the necessary decisions to resolve the ticket ASAP. 35 | ||all operational/ticketing/financial/performance reports run without any crash and contain all the relevant data. 36 | ||| 37 | | Business | customers should be satisfied with their experience with the ticketing system. | 38 | || changes in ticketing system should be carried out at minimal cost | 39 | || ticketing system should be available for a reasonable amount of time every day | 40 | ||| 41 | | Call center staff |system should be available during the work hours for handling any customer queries. 42 | ||to handle the customers' queries, system should be able to respond as quickly as possible with the accurate data. 43 | ||| 44 | | Development team |changes / new features can be done at minimum cost 45 | ||upgrades should be smooth, with minimum or reasonable downtime 46 | ||after deployment, system should remain stable with no crashes. 47 | ||| 48 | ||| 49 | -------------------------------------------------------------------------------- /2. Solution/RisksAndSensitivePoints.md: -------------------------------------------------------------------------------- 1 | # Risks 2 | 3 | ## Business related risks 4 | 5 | * The Customer UI is down - loss of customer/revenue 6 | * Option: Multiple DNS entry points, with load balancer cluster. 7 | * The Customer UI is up but shows either stale data or partial service/s is down - possible loss of customer 8 | * Option 1: leave to manual process with Helpdesk 9 | * Option 2: identify what exactly is stale and what parts still work, maybe tickets can be entered/processed in some reduced form and enriched by the Helpdesk later - notify the customer later 10 | * Option 3: if the customer not in a hurry, allow setting up a notification when the UI is fully back 11 | * The Customer can't find his particular model in the list of supported modules - loss of customer/revenue 12 | * Option 1: leave to manual process with Helpdesk 13 | * Option 2: allow entering customer's model as free form, to be resolved by experts/helpdesk, with a related warning 14 | * If resolved, the model is added to the list of supported models 15 | * The customer paid but the data is stale so he can't enter the ticket - loss of customer/revenue 16 | * Option 1: leave to manual process with Helpdesk - still possible loss 17 | * Option 2: Allow customers enter tickets even if they haven't paid, to resole the payment status later with helpdesk 18 | * Risk of DDOS with unactionable tickets 19 | * The ticket is in pending state for too long (no assigned Expert) - loss of customer/revenue 20 | * Option 1: after configured pending time, Helpdesk/Admin/Manager decide with Experts directly who's gonna action 21 | * Option 2: automatically raise the Expert fee for taking long pending tickets 22 | * Risk of abuse by Experts 23 | * Option 3: add a notion of "urgent" ticket, with additional fee on Customer 24 | * Risk of abuse by Experts 25 | * For a ticket, there are available experts but their available timeslots don't match any of Customer's timeslots 26 | * Option 1: the ticket will stay in a pending state, use the pending ticket process 27 | * Option 2: Let Customer know about available Experts' slots, maybe the Customer will adjust his slots to be serviced earlier/faster 28 | 29 | * The Expert is on his way to serve the Customer and there is an accident (car broken) that makes it impossible for the Expert to come in time or to come at all 30 | * Option 1: notify the Admin/Manager to quickly find another Expert 31 | * Option 2: have an automated way (in the Expert UI) for the affected Expert to ask other Experts to substitute for him (if no Experts reply, back to Option 1) 32 | * Option 3: 33 | 1. Add status to each visit, one/more visits is one ticket 34 | * The Expert comes but the Customer is not at home and can't be contacted 35 | * Option 1: wait for the whole agreed timeslot 36 | * Option 2: wait for configured time (e.g. 15 min) then be available for another task, ticket flagged 37 | * The Expert comes but finds the model different from what's in the ticket 38 | 39 | * The Expert comes but finds the he needs to purchase spare parts 40 | Possible, but Out Of Scope 41 | * The Expert comes but can't finish the job in time 42 | * Option 1: schedule another visit 43 | * Option 2 (if later tickets are not affected): let him continue 44 | * Option 3 NOT an option (if later tickets are affected): find substitution for a later ticket (unlcear what's with the expert fee for the lost later ticket - probably depends on the reason) 45 | * The Expert comes but finds the appliance irrepairable 46 | 47 | ## Technical Risks 48 | 49 | * With microservice architecture, any of the services can go down - it shouldn't bring the whole system down. Additional means like caching can be used to reduce mutual availability requirements. 50 | * When the system is distributed, the transactions become distributed as well. Integrity can be difficult to ensure. 51 | * Third-party services (messaging, maps) can go down - need to think of alternatives. If we don't have contracts with those services for the company consumption, and rely on users to use the services themselves (e.g. their own Google Maps on their devices), then we need to think of backup systems (e.g. Yahoo Maps). Also, some services might classify the usage as corporate and switch off or pin down. 52 | * Migration fails - should be ready to switch back to the existing system. 53 | * (Handled by Message Broker) Messages in queues lost because queues are down together with the messages - queued msgs should be stored in at least 2 places (maybe primary and backup) 54 | -------------------------------------------------------------------------------- /1. Problem Statement/BusinessGoalAndScope.md: -------------------------------------------------------------------------------- 1 | # System overview 2 | 3 | ## Business goal 4 | 5 | Business goal: 6 | 7 | To improve the current Sysops squad by redesign or process re-engineering to make the system consistence and reliable 8 | 9 | Fix the issues with Customer Ticket fallout. 10 | Availability issue 11 | Unresponsiveness 12 | Make the application consistentant and exit gracefully 13 | Fix the wrong assignement of Tickets 14 | 15 | ## Background: 16 | 17 | Things have not been good with the Sysops Squad lately. The current trouble ticket system is a large monolithic application that was developed many years 18 | ago. Customers are complaining that consultants are never showing up due to lost tickets, and often times the wrong consultant shows up to fix something 19 | they know nothing about. Customers and call-center staff have been complaining that the system is not always available for web-based or call-based problem 20 | ticket entry. Change is difficult and risky in this large monolith - whenever a change is made, it takes too long and something else usually breaks. Due to 21 | reliability issues, the monolithic system frequently “freezes up” or crashes - they think it’s mostly due a spike in usage and the number of customers using the 22 | system. If something isn’t done soon, Penultimate Electronics will be forced to abandon this very lucrative business line and fire all of the experts (including 23 | you, the architect). 24 | 25 | 26 | ### System features and responsibilities 27 | 28 | #### User Maintenance, Expert Profile 29 | Sysops squad experts are added and maintained in the system through an administrator, who enters in their locale, availability, and skills. 30 | 31 | #### Customer Profile, Support Contract, Billing System and History 32 | Customers who have purchased the support plan can enter a problem ticket using the sysops squad website. Customer registration for the support 33 | service is part of the system. The system bills the customer on an annual basis when their support period ends by charging their registered credit card. 34 | 35 | #### Ticket maintenance, Ticket Route and assign 36 | Once a trouble ticket is entered in the system, the system then determines which sysops squad expert would be the best fit for the job based on skills, 37 | current location, service area, and availability (free or currently on a job). 38 | 39 | #### Notification, Ticket Route and assign 40 | The sysops squad expert is then notified via a text message that they have a new ticket. Once this happens an email or SMS text message is sent to the 41 | customer (based on their profile preference) that the expert is on their way. 42 | 43 | #### User Interface, Ticket Shared, KB Search 44 | The sysops squad expert then uses a custom mobile application on their phone to access the ticketing system to retrieve the ticket information and 45 | location. The sysops squad expert can also access a knowledge base through the mobile app to find out what things have been done in the past to fix the 46 | problem. 47 | 48 | #### KB Maintenance 49 | Once the sysops squad expert fixes the problem, they mark the ticket as “complete”. The sysops squad expert can then add information about the 50 | problem and fix to the knowledge base. 51 | 52 | #### Survay, Notification 53 | After the system receives notification that the ticket is complete, the system send an email to the customer with a link to a survey which the customer then 54 | fills out. 55 | 56 | ### Numbers/Metrics 57 | Volume of request 58 | - 100 per day, 59 | - 1000 per month, 60 | No. of uses - 1000s 61 | 62 | ### Users 63 | 64 | #### Customers 65 | The customer registers for the Sysops Squad service, maintains their customer profile, support contracts, and billing information. Customers 66 | enter problem tickets into the system, and also fill out surveys after the work has been completed. 67 | 68 | #### Administrators 69 | The administrator user maintains the internal users of the system, including the list of experts and their corresponding skillset, location, and availability. The administrator also manages all of the billing processing for customers using the system, and maintains static reference data (such as supported products, name-value pairs in the system, and so on). 70 | 71 | #### Technician 72 | Experts are assigned problem tickets and fix problems based on the ticket. They also interact with the knowledge base to search for solutions to customer problems and also enter notes about repairs. 73 | 74 | #### Managers 75 | The manager keeps track of problem ticket operations and receives operational and analytical reports about the overall Sysops Squad problem ticket system. 76 | 77 | 78 | ## Out of scope 79 | 80 | UI/UX Design 81 | -------------------------------------------------------------------------------- /3. Perspectives/Readme.md: -------------------------------------------------------------------------------- 1 | # Important scenarios 2 | ## Ticket creation 3 | 4 | ### Stakeholders 5 | * Customer - Interested in entering a ticket the fastest and easiest way, get it assigned to experts, and executed in a timely fashion 6 | * Expert - Interested in receiving available tickets to execute and be paid for those 7 | * Helpdesk/Admin - Interested in most automation and error-proof process 8 | 9 | ### Risks 10 | 11 | * Login service can be down (not shown here) - it's duplicated under load-balancer cluster 12 | * Appliance service can be down - the Customer can't see a list of models and pick his model. Solutions: 13 | * The Customer UI service has a cache of appliances, so the problem is only if there is a new model not yet propagated to the cache. 14 | * For that unlikely scenario, Customer can always use a free-text to specify a model 15 | * Payment service can be down - we don't know if Customer has paid his monthly subscription charge. Solutions: 16 | * Cache for the paid customers in the Ticket service - only the the unlikely case that the Customer paid while the Payment data service was down gives trouble 17 | * If the payment data is not in the cache - the ticket is still created but flagged for the Helpdesk attention who will either resolve the problem on the backend or contact the Customer and resolve the issue with them directly 18 | * Hard to implement a good algorithm to match Experts to tickets. Solution: 19 | * Let Experts see the available tickets matching their skillset, and match themselves to a ticket (we call it "UBER model"). This insentivises the Experts to reply to tickets faster and have more tickets done. 20 | * No Experts matched for a ticket. Solution: 21 | * Wait the match to happen within pre-configured time. If it takes too long, then 22 | * Helpdesk/Admin/Manager takes care of the communication with the available Experts to get the ticket assigned. 23 | 24 | ![Ticket creation](TicketCreation.png) 25 | 26 | 27 | ## Ticket-to-expert matching 28 | 29 | ### Stakeholders 30 | * Customer - Interested in having the ticket assigned to experts and executed in a timely fashion 31 | * Expert - Interested in receiving available tickets to execute and be paid for those 32 | * Helpdesk/Admin - Interested in most automation and error-proof process 33 | 34 | ### Risks 35 | 36 | * A ticket may require multiple visits, or a joint visit my multiple experts, or visits happening one right after another (e.g. plumber and electrician). Solution: 37 | * We introduce a new entity - Visit. A ticket has 1..N visits. Each visit can have multiple experts (which can be converted to multiple tickets in the backend), and there can be time dependency between them. 38 | * There can be specific patterns that are hard to match in one try. Solution: 39 | * Each Expert who would like to do a visit proposes himself to all the times that would work for him. There can be several Experts attaching themselves to a visit. 40 | * When a group of dependent visits is full, the Experts attachment is finalized based on 1) the earlist visit for a Customer; 2) time priority for Experts. At this time, a notification is sent both to the Customer and to the finalized Experts who will go to execute the visit. 41 | * There are available Experts but their time slots don't match the Customer's. Solution: 42 | * When visits are not matched for some time, Helpdesk contacts the Customer and explains the situation and proposes available timeslots of Experts. If the Customer agrees to the time change, the visit is updated and Experts for a visit are finalized. 43 | * This can be automated later. 44 | 45 | ![Ticket-to-expert matching](TicketToExpertMatching.png) 46 | 47 | 48 | ## Ticket execution (visits) 49 | 50 | ### Stakeholders 51 | * Customer - Interested in ticket executed in a timely fashion 52 | * Expert - Interested in executing a ticket without problems 53 | * Helpdesk/Admin - Interested in most automation and error-proof process 54 | 55 | ### Risks 56 | * Expert can't arrive to the Customer's place due to an accident. Solution: 57 | * If time permits and Expert is replaceable (i.e. any Expert of the desired skillset can come), Helpdesk finds and sends a replacement, otherwise notifies the Customer and reschedules the visit. 58 | * Expert arrived but can't enter (nobody's home). Solution: 59 | * Helpdesk contacts the Customer and resolves the situation (possibly rescheduling the visit) 60 | * Expert can't finish job in time. Solutions: 61 | * If more time needed and time permits (i.e. no later visits for other Customers affected), Expert stays longer and completes the visit. 62 | * If there is no time, or it's impossible to finish because the needed spare parts need to be bought, Customer schedules a new visit, possibly with a help of Helpdesk. 63 | 64 | ![Ticket execution (visits)](TicketExecution.png) 65 | -------------------------------------------------------------------------------- /2. Solution/MigrationApproach.md: -------------------------------------------------------------------------------- 1 | ### Migration Approach 2 | The current application is a monolith and in a hard to maintain state, where changes take a long time and are error prone. To mitigate the problems with current application, the proposed new application architecture consists of microservices that primarily communicate with each other asynchronously using a message bus. 3 | 4 | #### Pre-requisites 5 | - Automated regression test suite. 6 | 7 | #### Development 8 | During the development phase, 9 | - Development needs to be done for the components shown below in the Component Mapping table. 10 | - For every component, an automation test suite that tests all its functionality needs to be built. 11 | - Existing components which will get reused, need to be modified to work as microservices that use asynchronous communication and a separate database with data as shown in the Database Mapping section. For existing components, test suite needs to compare the test results for the old implementation with the new one and fixes need to go in, where appropriate. 12 | - Test suite should also cover the integration work flows between the microservices. 13 | - The entire application needs to be run in trial mode for select beta customers. 14 | 15 | #### Deployment 16 | The deployment of the new solution involves the following to be done during the off-business hours: 17 | 1. Stop the old application services. 18 | 1. Migrate the data from the database of the old application to the new microservice-specific databases. The table mapping details are given in the "Database Mapping" section below. 19 | 1. After the migration is complete, start the new microservices. 20 | 21 | #### Migration Sequence 22 | As described [here](MinimumViableProduct.md), migration will first target the ticket creation/updating flow so that the problem of lost tickets is resolved. Next, the ticket assignment flow will be targeted to ensure that the right experts pick up the tickets. At this point we would have the microservices - Ticket Service and Expert Shortlisting Service which target the most pressing problems of the existing application. The subequent micro services will be moved gradually in a series of MVPs. 23 | 24 | #### Existing to new Components Mapping 25 | 26 | | # | Existing Systems's Component | New System's Component | Action | 27 | |----|----|----|----| 28 | |1. | Login | Login Service | Rename of the service | 29 | |2. | Billing Payment | Billing Service | Merged into new service | 30 | |3. | Billing History | Billing Service | Merged into new service | 31 | |4. | Customer Notification | Notification Service | New Service for all notifications | 32 | |5. | Ticket Notify | Notification Service | New Service for all notifications | 33 | |6. | Survey Notify | Notification Service | New Service for all notifications | 34 | |7. | Customer Profile | Customer Service | Merged into new service | 35 | |8. | Support Contract | Customer Service | Merged into new service | 36 | |9. | Expert Profile | Expert Service | Rename of the service | 37 | |10. | KB Maintenance | KB Service | Merged into new service | 38 | |11. | KB Search | KB Service | Merged into new service | 39 | |12. | Reporting | Reporting Service | Rename of the service | 40 | |13. | Ticket Shared | Ticket Service | Merged into new service | 41 | |14. | Ticket Assign | Ticket Service | Changed ticketing business process -> ADR.003 | 42 | |15. | Ticket Route | Expert Shortlisting Service | Redesign functionality | 43 | |16. | Survey | Survey Service | Merged into new service | 44 | |17. | Survey Template | Survey Service | Merged into new service | 45 | |18. | User Maintenance | User Service | Rename of the service | 46 | 47 | ![Mapping of components](./img/ServiceMapping.png) 48 | 49 | #### Database Mapping 50 | | # | Existing DB Table | New DB | 51 | |----|----|----| 52 | |1. | ss.Customer | Customer DB, Billing DB, Ticket DB, Survey DB | 53 | |2. | ss.Customer_Notification | Notification DB, Customer DB | 54 | |3. | ss.Survey | Survey DB, Report DB | 55 | |4. | ss.Survey_Question | Survey DB | 56 | |5. | ss.Customer_Survey | Survey DB, Report DB | 57 | |6. | ss.Customer_Survey_Question | Survey DB | 58 | |7. | ss.Customer_Survey_Response | Survey DB | 59 | |8. | ss.Billing | Billing DB, Report DB | 60 | |9. | ss.Contract | Billing DB, Report DB | 61 | |10. | ss.Payment_Method | Billing DB | 62 | |11. | ss.Payment | Billing DB | 63 | |12. | ss.SysOps_User | Expert DB, Ticket DB, Notification DB, Knowledge Base DB, Report DB, Survey DB, Expert Shortlist DB, User DB | 64 | |13. | ss.Profile | Expert DB, Report DB, User DB | 65 | |14. | ss.Expert_Profile | Expert DB, Expert Shortlist DB, Report DB | 66 | |15. | ss.Expertise | Expert DB, Expert Shortlist DB, Report DB | 67 | |16. | ss.Location | Expert DB, Expert Shortlist DB | 68 | |17. | ss.Article | Knowledge Base DB, Report DB | 69 | |18. | ss.Tag | Knowledge Base DB | 70 | |19. | ss.Keyword | Knowledge Base DB | 71 | |20. | ss.Article_Tag | Knowledge Base DB | 72 | |21. | ss.Article_Keyword | Knowledge Base DB | 73 | |22. | ss.Ticket | Ticket DB, Report DB | 74 | |23. | ss.Ticket_Type | Ticket DB, Report DB | 75 | |24. | ss.Ticket_History | Ticket DB, Knowledge Base DB, Report DB | 76 | --------------------------------------------------------------------------------