검색 상세

Using machine learning for anomaly detection and recommending scalable architecture for performance enhancement in Nepal’s RMIS

초록/요약

Nepal’s Revenue Management Information System (RMIS) is the backbone for collecting government revenue. It is used daily by thousands of bank staff, government officials and public citizens, yet it slows down or fails at peak hours such as tax deadlines, quarter-end and year-end. These disruptions delay services and weaken trust in digital government. This study asks a practical question: can RMIS remain steady during heavy demand, and if so, how? We take a two-part approach. First, we analyze operational data like CPU and database load, plus application and web logs using machine-learning models to spot unusual behavior before it causes outages. Among the methods tested, Isolation Forest was particularly good at flagging irregular patterns early, shifting operations from reacting after failure to preventing it. Second, we run scalability tests with Apache JMeter to compare the current WebLogic setup with a Kubernetes deployment. WebLogic handled light traffic, but performance fell quickly under sustained pressure. Adding more WebLogic instances only delayed the bottlenecks and required constant manual tuning of threads and virtual servers. Kubernetes adapted better. It stayed stable with 500 concurrent users and continued to perform when the load doubled to 1,000, helped by automatic horizontal and vertical scaling and priority-based resource allocation. Linking anomalies to metrics made diagnosis faster. For example, Nginx upstream errors and WebLogic thread congestion aligned with performance drops, allowing quick root-cause identification. Overall, the findings show that combining anomaly detection with Kubernetes can make RMIS more reliable using tools that are affordable and scalable in low- and middle-income countries like Nepal. The same approach can strengthen other e-government platforms, including tax filing, land registration and passport services.

more

목차

1. Introduction 1
1.1. Background of the Study 1
1.2. Deployment Architecture of RMIS 2
1.3. Technological Shifts in Deployment 2
1.4. Problem Statement 3
1.5. Objectives & Goals 3
1.6. Research Questions 3
1.7. Impacts and Significance of the Study 4
2. Literature Review 5
2.1. Machine Learning Techniques for Anomaly Detection 5
2.1.1. Supervised Machine Learning Techniques 6
2.1.2. Unsupervised Machine Learning Techniques 7
2.1.3. Hybrid and Ensemble Techniques 9
2.2. Anomaly Detection in Time-Series Data 10
2.2.1. Classical and Statistical Techniques 10
2.2.2. Machine Learning-Based Approaches 11
2.2.3. Deep Learning-Based Techniques 11
2.2.4. Seasonal and Log-Based Time-Series Detection 12
2.2.5. Real-Time and Streaming Detection Systems 12
2.3. Application Performance Issues 13
2.3.1. Introduction to Application Performance 13
2.3.2. Common Causes of Performance Degradation 13
2.3.3. Challenges in Monitoring and Diagnosing Performance 14
2.3.4. Performance Testing and Automation 15
2.3.5. Real-Time Monitoring and Adaptive Management 15
2.3.6. Issues in Component-Based and Parallel Applications 16
2.4. Log Analysis in Anomaly Detection 17
2.4.1. Deep Learning Approaches in Log-Based Anomaly Detection 17
2.4.2. Advanced Fusion Models for Log Analysis 18
2.4.3. Automation and Scalability in Log Analysis 18
2.5. Benchmarking and Case Studies 19
2.5.3. Evaluating Anomaly Detection Models in Real-World Web Applications 20
2.5.4. Performance Benchmarking in Cloud and Traditional Server Platforms 20
2.5.5. Fault Injection and Robustness Assessment Frameworks 21
2.5.6. Industrial-Scale Case Studies: Facebook, Amazon, and Aadhaar 22
2.6. Cluster Architecture for High Availability and Load Balancing 22
2.6.1. Introduction to Cluster-Based Architectures 22
2.6.2. From Three-Tier to Two-Tier Architecture: A Cost-Efficient Shift 23
2.6.3. Load Balancing Techniques Using NGINX in Web Clusters 23
2.6.4. Fault Tolerance in Cluster Task Scheduling and Load Management 24
2.6.5. Scalability and Performance Optimization in Cluster Design 24
2.6.6. Performance Testing and Real-World Benchmarking 25
2.7. Google Borg and Kubernetes 25
2.7.1. Introduction to Cluster Orchestration Systems 25
2.7.2. The Legacy of Borg: A Proven Internal System 26
2.7.3. Kubernetes: Democratizing Cluster Management 26
2.7.4. Architectural Differences and Evolution 27
2.7.5. Dynamic Scaling and QoS Management 28
2.7.6. Firmament, Poseidon, and Scheduling Optimizations 28
2.8. Research Gap and Conclusion 29
2.8.1. Lack of Proactive Anomaly Detection in Public Sector Platforms 29
2.8.2. Absence of Real-Time Scaling Linked to Anomaly Detection 29
2.8.3. Limited Use of Hybrid and Multivariate Models in Government Systems 30
2.8.4. Lack of Cluster-Aware Anomaly Detection 30
2.8.5. Underexplored Peak-Load Performance Prediction 30
2.8.6. Limited Use of Log Analysis in Anomaly Detection 30
2.8.7. Missing Root Cause Analysis Combining Metrics and Logs 31
2.8.8. Scarcity of LMIC-Specific Benchmarking and Case Studies 31
2.8.9. Unexplored Modern Resilient Architecture like Kubernetes 31
2.8.10. Conclusion: Why This Study is Needed 31
3. Methodology 33
3.1. Research Process Steps: 33
3.1.1. Data Collection 33
3.1.2. Dataset Merging 33
3.1.3. Feature Relationship Analysis 34
3.1.4. Implementation of Anomaly Detection 34
3.1.5. Response to Anomaly Detection – Scaling the WebLogic Cluster 34
3.1.6. Verification of Performance Improvement 34
3.1.7. Proposal of Alternative Architectures 35
3.1.8. Simulation of Alternative Architecture in an Isolated Environment 35
3.1.9. Peak Load Simulation 35
3.1.10. Evaluation of Peak Load Handling Capability 35
3.1.11. Summary of Methodological Approach 35
3.2. Research Walkthrough 35
3.2.1. Dataset description 36
3.2.2. Summary of the dataset 37
3.2.3. Features ranked by their correlation with anomalies 37
3.2.4. Abnormal and Normal Events by Hour of Day 38
3.2.5. Mean absolute SHAP values 39
3.2.6. PCA – Visualizing Clusters with Anomalies: 40
3.3. Anomalies per month: 41
3.4. Log Analysis 42
3.4.1. Oracle DB Log Analysis: 42
3.4.2. Nginx Log Analysis 43
3.4.3. WebLogic Log Analysis 43
3.5. Exploring Alternate Architectures 44
3.6. Google Borg Cluster 45
3.7. Kubernetes Cluster 46
3.7.1. Kubernetes Cluster Architecture 47
3.8. Comparing WebLogic & Kubernetes Cluster 47
3.9. Environment Setup and Testing of WebLogic and Kubernetes Architectures 49
3.10. Overview of the Two Environments 50
3.10.1. WebLogic Environment 50
3.10.2. Kubernetes Environment 51
3.10.3. Hardware and Software Setup 52
3.11. Proposed Anomaly Detection Mechanism 56
3.11.1. Data Collection 57
3.11.2. Model Training 57
3.11.3. Real-Time Detection 58
3.11.4. System Architecture 59
3.11.5. Benefits 61
3.11.6. Limitations 61
3.11.7. Conclusion 62
4. Results and Discussion 63
4.1. Performance Evaluation of Kubernetes and WebLogic under Load Conditions 63
4.2. Load Generation: 63
4.3. Test Scenarios 64
4.3.1. Scenario 1: Low Load Test 64
4.3.2. Scenario 2: Medium Load Test 83
4.3.3. Scenario 3: High Load Test 104
4.4. Comparison between Kubernetes and WebLogic based on JMeter Result: 125
4.4.1. For 100 JMeter Users 125
4.4.2. For 500 JMeter Users: 128
4.4.3. For 1000 JMeter Users 132
4.4.4. Summary of JMeter Test Results 135
4.5. JVM Comparison of Kubernetes and WebLogic 139
4.5.1. JVM Performance with 100 JMeter Users 140
4.5.2. JVM Performance with 500 JMeter Users 140
4.5.3. JVM Performance with 1000 JMeter Users 141
4.5.4. Comparative Summary 142
4.5.5. Comparative Performance in Graphical Representation (Bar Chart) 143
4.5.6. Comparative Performance in Graphical Representation (Line Chart) 144
4.5.7. Trend Analysis of Kubernetes vs WebLogic JVM Performance 146
4.5.8. Conclusion 147
4.6. Horizontal Autoscaling: Kubernetes vs WebLogic 147
4.6.1. Kubernetes Horizontal Pod Autoscaling (HPA) 147
4.6.2. WebLogic Horizontal Scaling 148
4.6.3. Comparative Insights 149
4.6.4. Conclusion 149
4.7. Vertical Autoscaling: Kubernetes vs WebLogic 149
4.7.1. Kubernetes Vertical Pod Autoscaling (VPA) 149
4.7.2. WebLogic Vertical Scaling 150
4.7.3. Comparative Insights 151
4.7.4. Conclusion 151
4.8. Priority Class: Kubernetes vs WebLogic 151
4.8.1. Priority Class in Kubernetes 151
4.8.2. Priority Handling in WebLogic 152
4.8.3. Comparative Insights 152
4.8.4. Conclusion 153
5. Conclusion and Recommendations 154
5.1. Conclusion & Recommendations 154
5.2. Limitations 158
5.2.1. Limited scope of testing environments 158
5.2.2. Resource constraints in experimentation 159
5.2.3. Dependence on specific algorithms 159
5.2.4. Lack of long-term testing 159
5.2.5. Incomplete coverage of external dependencies 160
5.2.6. Limited focus on cybersecurity risks 160
5.2.7. Generalization challenges 160
5.2.8. Data limitations 160
5.2.9. Practical barriers to implementation 161
5.2.10. Focus on backend performance 161
5.2.11. Financial and policy constraints 161
5.3. Future Works 162
5.3.1. Expanding Anomaly Detection Models 162
5.3.2. Real-Time Deployment and Continuous Learning 162
5.3.3. Large-Scale Stress Testing with Kubernetes 162
5.3.4. Integration of Logs, Metrics, and Traces 163
5.3.5. Security and Cyber-Resilience 163
5.3.6. Cost-Benefit and Policy Analysis 163
6. Action Plan 164
6.1. Action Plan for Implementing Anomaly Detection and Kubernetes in RMIS 164
6.2. Phase 1: Laying the Foundation 165
6.2.1. Stakeholder Engagement 165
6.2.2. Current System Audit 165
6.2.3. Skills and Capacity Assessment 165
6.3. Phase 2: Building Technical Capability 166
6.3.1. Establishing a Test Environment 166
6.3.2. Developing Anomaly Detection Models 166
6.3.3. Kubernetes Pilot Deployment 166
6.4. Phase 3: Expanding and Integrating 167
6.4.1. Gradual Migration from WebLogic 167
6.4.2. Advanced Monitoring and Log Integration 167
6.4.3. Training and Knowledge Transfer 167
6.5. Phase 4: Institutionalization 168
6.5.1. Policy and Governance Integration 168
6.5.2. Sustainable Financing 168
6.5.3. Citizen-Centered Service Delivery 168
6.6. Phase 5: Long-Term Evolution 169
6.6.1. Continuous Improvement 169
6.6.2. Cross-Sector Replication 169
6.7. Timeline and Milestones 170
6.8. Risks and Mitigation 170
6.9. Conclusion 171
References 172
Appendix 1 180
Appendix 2 213

more
반출 Meta View 목록