数据库学习资料（2022-03-18 更新）
作者: 康凯森
日期: 2021-12-19
分类: OLAP
- 数据库总览
- 数据库领域博客专栏
- 数据库系统综述
- 数据库系统简介
- Amazon Aurora
- AnalyticDB
- Arrow
- Apache Calcite
- Apache Pulsar
- AresDB
- Ceph
- ChubaoFS
- ClickHouse
- CockroachDB
- CynosDB
- DB2
- Dremio
- Druid
- F1
- Flink
- FoundationDB
- Google BigQuery
- Google Dremel
- Google Procella
- Google Napa
- GreenPlum
- HAWQ
- HBase
- HiStore
- Hudi
- Hyper
- Ignite
- Impala
- Kudu
- Mysql
- Mesa
- Memsql
- OceanBase
- OmniSciDB
- Pinot
- PostgreSQL
- Presto
- Redshift
- SAP HANA
- Spanner
- SnappyData
- Snowflake
- Spark
- Splice machine
- TiDB
- Vertica
- VoltDB
- X-Engine
- Hologres
- C++ 语言
- 查询优化器
- 查询执行
- 向量化
- 查询编译
- 存储
- 数据导入
- 测试
- 事务
- Cloud-Native
- 性能优化
在 StarRocks 将近两年的时间里面，我们查询团队从零实现了向量化执行器，CBO 查询优化器，Pipeline 并行查询引擎，刷新国产 OLAP 数据库性能的里程碑，本文整理了我平时参考和学习的一些数据库资料，希望对大家有所帮助，也欢迎大家参与 StarRocks 开源社区。
本文章会努力持续更新，也欢迎大家一起贡献和修改。
TODO:
- [ ] 系统综述部分进行分类
- [ ] 每篇材料给出难度评级（入门，进阶，专业）
- [ ] 每篇材料给出必要性评级 （可选，必读）
- [ ] 每篇论文备注下核心观点，以及我们可以学习到什么
- [ ] 每个数据库系统架构的优点，缺点，以及我们可以学习到什么
数据库总览
书籍
- Designing Data-Intensive Applications ★★★★★
- Database Internals
- Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing
- Architecture of a Database System
大学课程
- CMU 15445 数据库基础课程 ★★★★★
- CMU 15721 数据库高级课程 ★★★★★
- Database System Concepts
- Parallel and Distributed Programming ★★★★★
系列教程
文章
How does a relational database work
数据库领域博客专栏
- PingCAP Paper reading
- 数据系统论文阅读小组
- 数据库内核月报
- CMU Database Group 视频 ★★★★★
- Clickhouse 社区分享的PPT
- 知乎 Presto 专栏
- Awesome Database Learning
- TIDB Blog
- PostgreSQL Blog
- Umbra DB 论文
数据库系统综述
数据库系统简介
Amazon Aurora
AnalyticDB
AnalyticDB: Real-time OLAP Database System at Alibaba Cloud
Arrow
Apache Arrow: In Theory, In Practice
Apache Calcite
Apache Pulsar
AresDB
Introducing AresDB: Uber’s GPU-Powered Open Source, Real-time Analytics Engine
Ceph
Ceph: A Scalable, High-Performance Distributed File System
ChubaoFS
CFS: A Distributed File System for Large Scale Container Platforms
ClickHouse
The Secrets of ClickHouse Performance Optimizations
CockroachDB
CockroachDB: The Resilient Geo-Distributed SQL Database.pdf)
CynosDB
DB2
DB2 with BLU Acceleration: So Much More than Just a Column Store
Dremio
Using Apache Arrow, Calcite and Parquet to build a Relational Cache
Druid
Druid: A Real-time Analytical Data Store
F1
F1 Query: Declarative Querying at Scale
Flink
Apache Flink™: Stream and Batch Processing in a Single Engine
FoundationDB
FoundationDB: A Distributed Unbundled Transactional Key Value Store
Google BigQuery
- A Deep Dive Into Google BigQuery Architecture
- BigQuery under the hood
- Big Metadata: When Metadata is Big Data
Google Dremel
Dremel: Interactive Analysis of Web-Scale Datasets
Google Procella
Procella: Unifying serving and analytical data at YouTube
Google Napa
Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google
GreenPlum
Why Greenplum is the best compared with others
HAWQ
HAWQ: A Massively Parallel Processing SQL Engine in Hadoop
HBase
HiStore
Hudi
Apache Hudi Design And Architecture
Hyper
Ignite
Impala
Impala: A Modern, Open-Source SQL Engine for Hadoop
Kudu
Kudu: Storage for Fast Analytics on Fast Data
Mysql
Mesa
Mesa: Geo-Replicated, Near Real-Time, Scalable Data Warehousing
Memsql
OceanBase
OmniSciDB
Pinot
Pinot: Realtime OLAP for 530 Million Users
PostgreSQL
Presto
Redshift
Amazon Redshift and the Case for Simpler Data Warehouses
Data Warehousing in the Cloud: Amazon Redshift vs Microsoft Azure SQL
SAP HANA
SAP HANA: A Data Platform for Enterprise Applications Purpose Built for Modern Hardware ★★★★★
SAP HANA: A Data Platform for Enterprise Applications Purpose Built for Modern Hardware 视频
Spanner
Spanner: Google’s Globally-Distributed Database
SnappyData
SnappyData: A Unified Cluster for Streaming,Transactions, and Interactive Analytics
Snowflake
The Snowflake Elastic Data Warehouse
Building An Elastic Query Engine on Disaggregated Storage
Spark
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Spark SQL: Relational Data Processing in Spark
Splice machine
Splice Machine – An HTAP DB at Scale
TiDB
Tidb a raft based HTAP database
Vertica
The Vertica Analytic Database: C-Store 7 Years Later
VoltDB
X-Engine
X-Engine: An Optimized Storage Engine for Large-scale E-commerce Transaction Processing
Hologres
Alibaba Hologres: A Cloud-Native Service for Hybrid Serving/Analytical Processing
C++ 语言
- https://www.learncpp.com/ 比较系统，通俗易懂，适合入门时系统学习，也可以在忘记某个细节时查一查
- A Tour of C++ ★★★★★ 如果有其他编码语言基础，建议直接阅读此书入门 C++
- Build your own Database 介绍了数据库开发需要的 C++ 知识点，里面的知识点搞懂后，就可以开始用 C++ 开发数据库了
- https://isocpp.org/faq ★★★★★ 对某个 C++ 细节点搞不清楚时，可以进行参考
- Programming Notes for Professionals books Stack Overflow C++ 问题高频，经典问题的系统整理，在对某个 C++ 细节点搞不清楚时，可以进行参考
- CppCoreGuidelines C++ 编程规范和最佳实践，有空了可以系统过一遍，不过每个细节不用都记住，IDE 有提供对应的插件，可以在你编码时进行提醒
- CppCon 历年会议 PPT ★★★★★ 有很多关于 C++ 设计，语法，工具，性能优化，数据结构的精彩分享， 感觉自己 C++ 入门后，可以每天抽空看一个小分享
- Effective C++ 系列：Effective 系列有4本书，自己时间充裕的话，可以阅读全书，时间紧张的话建议直接阅读网上大家整理好的笔记
编程语言是工具，工具重在使用，所以学习编程语言和学习英语一样，提升最快的方式就是多使用，多实践，不需要等到对语言完全熟悉或者精通后才开始使用，实践中遇到了某个点不清楚，就一个点一个点逐一突破。
查询优化器
论文
- Efficiency in the Columbia Database Query Optimizer ★★★★★
- Orca: A Modular Query Optimizer Architecture for Big Data
- The Volcano Optimizer Generator: Extensibility and Efficient Search
- The Cascades Framework for Query Optimization
- The Complete Story of Joins (in HyPer)
- Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources
- Spark SQL: Relational Data Processing in Spark
- An Overview of Query Optimization in Relational Systems
- The MemSQL Query Optimizer: A modern optimizer for real-time analytics in a distributed database
- How Good Are Query Optimizers, Really? ★★★★★
- Optimization of Common Table Expressions in MPP Database Systems
- Optimizing Queries over Partitioned Tables in MPP Systems
- Adaptive Optimization of Very Large Join Queries
- Unnesting Arbitrary Queries
- Inside the SQL Server Query Optimizer
- Exploiting Upper and Lower Bounds in Top-Down Query Optimization
文章
- Introduction to Presto Cost-Based Optimizer
- CMU 数据库 Optimizer
- Deep Dive into Spark SQL’s Catalyst Optimizer
- ClustrixDB Query Optimizer
- OceanBase查询优化器的设计之道和工程实践哲学
- How We Built a Cost-Based SQL Optimizer
- 优化器技术论文学习
查询执行
论文
- Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age ★★★★★
- Expression Templates Revisited: A Performance Analysis of the Current ET Methodology
- Is Parallel Programming Hard, And, If So, What Can You Do About It?
- Push vs. Pull-Based Loop Fusion in Query Engines
PPT
- A Deep Dive into Query Execution Engine of Spark SQL
- ClickHouse 执行模型
- Understanding Task Scheduling Algorithms
- Smart Expression Templates
向量化
论文
- Vectorization vs. Compilation in Query Execution
- MonetDB/X100: Hyper-Pipelining Query Execution
- Rethinking SIMD Vectorization for In-Memory Databases
- Interleaved Multi-Vectorizing
- Balancing vectorized query execution with bandwidth-optimized storage
- Towards Practical Vectorized Analytical Query Engines
- Rethinking SIMD Vectorization for In-Memory Databases
- BIPie: Fast Selection and Aggregation on Encoded Data using Operator Specialization
- SAHA: A String Adaptive Hash Table for Analytical Databases
课程
文章 & PPT
- Vectorized Query Processing Using Apache Arrow
- http://0x80.pl/ 专注于 SIMD 编程，有大量实践文章。
- Daniel Lemire's blog 专注于高性能计算，有大量优秀文章。
- Data-Parallel Execution using SIMD Instructions
- Crunching Numbers with AVX and AVX2
- SSE & AVX Vectorization
- Basics of SIMD Programming
- 最快的 integer-parsing 实现
- 使用 AVX 系列指令集进行向量化 文末的链接可以看下
- TIDB 10x Performance Improvement for Expression Evaluation Made Possible by Vectorized Execution and the Community
- 40x faster hash joiner with vectorized execution
- Scatter Instruction
- GCC Autovectorization
- Requirements for Vectorizable Loops
- SIMD Programming
- Why are unrolled loops faster
- SIMD at Insomniac Games (How we do the shuffle)
- Software optimization resources
- Vectorization with Compilers (C/C++)
- SIMD Vectorized Hashing for Grouped Aggregation
- Practical SIMD Programming
查询编译
论文
- Efficiently Compiling Efficient Query Plans for Modern Hardware
- Relaxed Operator Fusion for In-Memory Databases: Making Compilation, Vectorization, and Prefetching Work Together At Last
- Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask
- Adaptive Execution of Compiled Queries
- Runtime Code Generation in Cloudera Impala
- LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation
- Generating code for holistic query evaluation
- Low-Latency Compilation of SQL Qeries to Machine Code
- Permutable Compiled Queries: Dynamically Adapting Compiled Queries without Recompiling
课程
文章 & PPT
- Using LLVM to Accelerate Processing of Data in Apache Arrow
- Apache Spark as a Compiler: Joining a Billion Rows per Second on a Laptop
- Code Generation: The Inner Sanctum Of Database Performance
- JIT-Compiling SQL Queries in PostgreSQL Using LLVM
- JIT C++ compiler with LLVM
- 查询编译综述
- Overhead of a Generalized Query Execution Engine
存储
论文
- Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation
- 基于 Tile 连接 Row-Store 和 Column-Store
- PAX：一个 Cache 友好高效的行列混存方案
- VLDB 2009 Tutorial on Column-Stores ★★★★★
- The Design and Implementation of Modern Column-Oriented Database Systems ★★★★★
- Main Memory Database Systems
- WiscKey: Separating Keys from Values in SSD-conscious Storage
- Facebook’s Tectonic Filesystem: Efficiency from Exascale
- Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats
文章
- X-Engine的In-Memory读性能优化
- Database Compression
- Understanding SQL Server Memory-Optimized Tables Hash Indexes
- Apache CarbonData:New high performance data format for faster data analysis
- Apache Parquet
- Parquet Encoding
- Apache ORC File
- 2018 存储技术热点与趋势总结
- 2020 存储技术热点与趋势总结
- 设计SSD友好的应用
- 非易失性内存在阿里生产环境的应用：Tair NVM实践总结
- MemSQL SingleStore – And Then There Was One
数据导入
- Instant Loading for Main Memory Databases
- Speculative Distributed CSV Data Parsing for Big Data Analytics
测试
- Quantifying TPC-H Choke Points and Their Optimizations
- Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction
- Automated SQL Query Generation for Systematic Testing of Database Engines
- QAGen: Generating Query-Aware Test Databases
- Squirrel: Testing Database Management Systems with Language Validity and Coverage Feedback
- Testing Database Engines via Pivoted Query Synthesis
- Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction
- Finding Bugs in Database Systems via Query Partitioning
事务
- A Critique of ANSI SQL Isolation Levels
- Transaction Processing: Concepts and Techniques
- Granularity of Locks and Degrees of Consistency in a Shared Data Base
- (Percolator) Large-scale Incremental Processing Using Distributed Transactions and Notifications
- (Omid) Omid, Reloaded: Scalable and Highly-Available Transaction Processing
- (2PC) Database System Concepts.
- (2PC) Database Systems The Complete Book
- (OCC) On Optimistic Methods for Concurrency Control
- (TicTok) TicToc: Time Traveling Optimistic Concurrency Control
- (Silo) Speedy Transactions in Multicore In-Memory Databases
- (Hekaton) High-Performance Concurrency Control Mechanisms for Main-Memory Databases
- Deferred Action Framework
Cloud-Native
待完善
- What Serverless Computing Is and Should Become: The Next Phase of Cloud Computing
- Starling: A Scalable Query Engine on Cloud Function Services
- Bringing Cloud-Native Storage to SAP IQ
- cloud data lakes for dummies
性能优化
- Intel® 64 and IA-32 Architectures Optimization Reference Manual
- performance_mattersemery_bergercppcon_2020
- Denis Bakhvalov - Performance Analysis and Tuning on Modern CPUs
- A Top-Down Method for Performance Analysis and Counters Architecture ★★★★★
- Systems Performance Enterprise and the Cloud by Brendan Gregg
- BPF Performance Tools
- Computer Architecture: A Quantitative Approach
Profile 工具
- https://godbolt.org/
- https://quick-bench.com/
- vtune-profiler ★★★★★
- https://github.com/andikleen/pmu-tools/wiki/toplev-manual
- https://github.com/facebookincubator/BOLT
- https://github.com/plasma-umass/coz
- Profiling: Understand Your Application
- Benchmarking C++ Code
CPU 微架构
- Understanding CPU Microarchitecture to Increase Performance
- CPU的制造和概念
- Skylake微架构(Microarchitecture)剖析
- Computer Arch
CPU Cache
- https://www.stardog.com/blog/writing-cache-friendly-code/
- Memory part 5: What programmers can do
- Machine Architecture: Things Your Programming Language Never Told You
- Advanced Caching Techniques
- An Overview of Cache Optimization Techniques and Cache-Aware Numerical Algorithms
- What You Must Know about Memory,Caches, and Shared Memory ★★★★★
如果学习了某部分理论，却不知如何实践，欢迎参与 StarRocks 开源社区，上面所有的理论在 StarRocks 中都有可以实践的地方，欢迎大家的 Star, Issue 和 PR。
《OLAP 性能优化指南》欢迎 Star&共建
欢迎关注微信公众号
