1

01

引言

實現并發編程有進程，線程，IO多路復用的方式。(并發和并行我們這里不區分，如果CPU是多核的，可能在多個核同時進行，我們叫并行，如果是單核，需要排隊切換，我們叫并發)。

1.1

進程和線程的區別

進程是計算機資源分配的最小單位，進程是對處理器資源(CPU)，虛擬內存(1)的抽象，虛擬內存是對主存資源 (Memory) 和文件(2)的抽象，文件是對 I/O 設備的抽象。

虛擬內存是操作系統初始化后內部維護的一個程序加載空間，對于32位操作系統來說，也就是寄存器有32位的比特長度，虛擬內存中每個字節都有一個內存地址，內存地址的指針長度為32位(剛好是寄存器可以存放的位數)，算下來2的32次，剛好可以存放4G左右的字節，所以在32位的操作系統上，你的8G內存條只有50%的利用率，所以現在都是64位的操作系統。

其中，CPU，Memory，I/O設備就是我們所說的CPU核，內存，硬盤。

線程是計算機調度的最小單位，也就是CPU大腦調度的最小單位，同個進程下的線程可以共享同個進程分配的計算機資源。同個進程下的線程間切換需要CPU切換上下文，但不需要創建新的虛擬內存空間，不需要內存管理單元切換上下文，比不同進程切換會顯得更輕量。

總上所述，實際并發的是線程。首先，每個進程都有一個主線程，因為線程是調度的最小單位，你可以只有一個線程，但是你也可以創建多幾個線程，線程調度需要CPU來切換，需要內核層的上下文切換，如果你跑了A線程，然后切到B線程，內核調用開始，CPU需要對A線程的上下文保留，然后切到B線程，然后把控制權交給你的應用層調度。進程切換也需要內核來切換，因為從C進程的主線程切換到D進程的主線程。。。

1.2

進程間通信和線程間通信

那么進程間要通訊呀，而且他們資源不共享，這個時候需要用 IPC(Inter-Process Communication，進程間通信)，常用的有信號量，共享內存，套接字，實際百度上說有六種耶。

而同個進程的多個線程共享資源，通訊起來比進程容易多了，因為它們共享了虛擬內存的空間，直接就可以讀取內存，現在很多 Python，Java 等編程語言都有這種線程庫實現。

至于 IO 多路復用，其實就是維持一個線程隊列，然后讓一個線程或多個線程，去隊列里面拿任務去完成。為什么呢？因為線程的數量是有限的，而且線程間通訊需要點資源，內核也要頻繁切換上下文，干脆就弄一個池，有任務就派個小弟出去。

只有一個線程的 IO 多路復用，典型的就是 Redis 和 Nodejs 了，根本不需要切換上下文，一個線程走天下。而多個線程的 IP 多路復用，就是 Golang 協程的實現方式了，協程，自己管理線程，把線程控制到一定的數量，然后構造一個規則狀態機來調度任務。

1

02?

Golang 協程?

無論是一個進程下的多個線程，還是不同進程，還是不同進程下的線程，切換時都需要損耗資源，浪費一些資源。所以 Golang 有協程這種東西，就是在語言內部管理自己的一些線程，合理的調度方式，使得線程不那么頻繁的切換。

Golang 語言的調度器其實就是通過使用數量合適的線程并在每一個線程上執行更多的工作來降低操作系統和硬件的負載。

2.1

調度器數據結構

Golang調度器有三個主要數據結構。

M，操作系統的線程，被操作系統管理的，原生線程。

G，協程，被Golang語言本身管理的線程，該結構體中包含一些指令或者調度的信息。

P，調度的上下文，運行在M上的調度器。

也就是說，Golang 使用?P?來對?M?進行管理，延伸出?G?的概念。

2.2

G?假線程

Goroutine，也就是?G，只存在于 Go 語言運行時，是對實際操作系統線程的映射，一般是?M:1?映射。也就是說，可能 5? 個G，其實真實情況只有1個M。Golang幫你做了調度，幫你進行了抽象。

數據結構定義可以在此查看：/src/runtime/runtime2.go

type g struct { // Stack parameters. // stack describes the actual stack memory: [stack.lo, stack.hi). // stackguard0 is the stack pointer compared in the Go stack growth prologue. // It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption. // stackguard1 is the stack pointer compared in the C stack growth prologue. // It is stack.lo+StackGuard on g0 and gsignal stacks. // It is ~0 on other goroutine stacks, to trigger a call to morestackc (and crash). stack stack // offset known to runtime/cgo stackguard0 uintptr // offset known to liblink stackguard1 uintptr // offset known to liblink _panic *_panic // innermost panic - offset known to liblink _defer *_defer // innermost defer m *m // current m; offset known to arm liblink sched gobuf syscallsp uintptr // if status==Gsyscall, syscallsp = sched.sp to use during gc syscallpc uintptr // if status==Gsyscall, syscallpc = sched.pc to use during gc stktopsp uintptr // expected sp at top of stack, to check in traceback param unsafe.Pointer // passed parameter on wakeup atomicstatus uint32 stackLock uint32 // sigprof/scang lock; TODO: fold in to atomicstatus goid int64 schedlink guintptr waitsince int64 // approx time when the g become blocked waitreason waitReason // if status==Gwaiting preempt bool // preemption signal, duplicates stackguard0 = stackpreempt paniconfault bool // panic (instead of crash) on unexpected fault address preemptscan bool // preempted g does scan for gc gcscandone bool // g has scanned stack; protected by _Gscan bit in status gcscanvalid bool // false at start of gc cycle, true if G has not run since last scan; TODO: remove? throwsplit bool // must not split stack raceignore int8 // ignore race detection events sysblocktraced bool // StartTrace has emitted EvGoInSyscall about this goroutine sysexitticks int64 // cputicks when syscall has returned (for tracing) traceseq uint64 // trace event sequencer tracelastp puintptr // last P emitted an event for this goroutine lockedm muintptr sig uint32 writebuf []byte sigcode0 uintptr sigcode1 uintptr sigpc uintptr gopc uintptr // pc of go statement that created this goroutine ancestors *[]ancestorInfo // ancestor information goroutine(s) that created this goroutine (only used if debug.tracebackancestors) startpc uintptr // pc of goroutine function racectx uintptr waiting *sudog // sudog structures this g is waiting on (that have a valid elem ptr); in lock order cgoCtxt []uintptr // cgo traceback context labels unsafe.Pointer // profiler labels timer *timer // cached timer for time.Sleep selectDone uint32 // are we participating in a select and did someone win the race? // Per-G GC state // gcAssistBytes is this G's GC assist credit in terms of // bytes allocated. If this is positive, then the G has credit // to allocate gcAssistBytes bytes without assisting. If this // is negative, then the G must correct this by performing // scan work. We track this in bytes to make it fast to update // and check for debt in the malloc hot path. The assist ratio // determines how this corresponds to scan work debt. gcAssistBytes int64}

結構?G?定義了一個字段?atomicstatus，表示當前這個協程的狀態：

// defined constantsconst ( // G status // // Beyond indicating the general state of a G, the G status // acts like a lock on the goroutine's stack (and hence its // ability to execute user code). // // If you add to this list, add to the list // of "okay during garbage collection" status // in mgcmark.go too. // // TODO(austin): The _Gscan bit could be much lighter-weight. // For example, we could choose not to run _Gscanrunnable // goroutines found in the run queue, rather than CAS-looping // until they become _Grunnable. And transitions like // _Gscanwaiting -> _Gscanrunnable are actually okay because // they don't affect stack ownership. // _Gidle means this goroutine was just allocated and has not // yet been initialized. // 剛剛被分配并且還沒有被初始化 _Gidle = iota // 0 // _Grunnable means this goroutine is on a run queue. It is // not currently executing user code. The stack is not owned. // 沒有執行代碼、沒有棧的所有權、存儲在運行隊列中 _Grunnable // 1 // _Grunning means this goroutine may execute user code. The // stack is owned by this goroutine. It is not on a run queue. // It is assigned an M and a P. // 可以執行代碼、擁有棧的所有權，被賦予了內核線程 M 和處理器 P _Grunning // 2 // _Gsyscall means this goroutine is executing a system call. // It is not executing user code. The stack is owned by this // goroutine. It is not on a run queue. It is assigned an M. // 正在執行系統調用、擁有棧的所有權、沒有執行用戶代碼，被賦予了內核線程 M 但是不在運行隊列上 _Gsyscall // 3 // _Gwaiting means this goroutine is blocked in the runtime. // It is not executing user code. It is not on a run queue, // but should be recorded somewhere (e.g., a channel wait // queue) so it can be ready()d when necessary. The stack is // not owned *except* that a channel operation may read or // write parts of the stack under the appropriate channel // lock. Otherwise, it is not safe to access the stack after a // goroutine enters _Gwaiting (e.g., it may get moved). // 由于運行時而被阻塞，沒有執行用戶代碼并且不在運行隊列上，但是可能存在于 Channel 的等待隊列上 _Gwaiting // 4 // _Gmoribund_unused is currently unused, but hardcoded in gdb // scripts. // 暫無作用 _Gmoribund_unused // 5 // _Gdead means this goroutine is currently unused. It may be // just exited, on a free list, or just being initialized. It // is not executing user code. It may or may not have a stack // allocated. The G and its stack (if any) are owned by the M // that is exiting the G or that obtained the G from the free // list. // 沒有被使用，沒有執行代碼，可能有分配的棧 _Gdead // 6 // _Genqueue_unused is currently unused. // 暫無作用 _Genqueue_unused // 7 // _Gcopystack means this goroutine's stack is being moved. It // is not executing user code and is not on a run queue. The // stack is owned by the goroutine that put it in _Gcopystack. // 棧正在被拷貝、沒有執行代碼、不在運行隊列上 _Gcopystack // 8 // _Gscan combined with one of the above states other than // _Grunning indicates that GC is scanning the stack. The // goroutine is not executing user code and the stack is owned // by the goroutine that set the _Gscan bit. // // _Gscanrunning is different: it is used to briefly block // state transitions while GC signals the G to scan its own // stack. This is otherwise like _Grunning. // // atomicstatus&~Gscan gives the state the goroutine will // return to when the scan completes. // 為了更友好，把0加上些前綴 _Gscan = 0x1000 _Gscanrunnable = _Gscan + _Grunnable // 0x1001 _Gscanrunning = _Gscan + _Grunning // 0x1002 _Gscansyscall = _Gscan + _Gsyscall // 0x1003 _Gscanwaiting = _Gscan + _Gwaiting // 0x1004)

主要有 _Grunnable、_Grunning、_Gsyscall 和 _Gwaiting 四個狀態：

_Grunnable 沒有執行代碼、沒有棧的所有權、存儲在運行隊列中_Grunning 可以執行代碼、擁有棧的所有權，被賦予了內核線程 M 和處理器 P_Gsyscall 正在執行系統調用、擁有棧的所有權、沒有執行用戶代碼，被賦予了內核線程 M 但是不在運行隊列上_Gwaiting 由于運行時而被阻塞，沒有執行用戶代碼并且不在運行隊列上，但是可能存在于 Channel 的等待隊列上

上面進行抽象，Goroutine?可能在等待某些滿足條件，處于等待中，當滿足條件時會變成可運行狀態，等待被調度到真實的線程?M，如果伙伴太多可能需要等很久，等到了會進入運行中，表示正在某個M上執行。

2.3

M?真線程

Golang 默認情況下，調度器可以創建很多線程，但是最多只有?gomaxprocs?個M真線程能真正正常運行。

allp []*p // len(allp) == gomaxprocs; may change at safe points, otherwise immutable 每一個真線程M都會被綁定一個調度器P allpLock mutex // Protects P-less reads of allp and all writes gomaxprocs int32

通常情況下，gomaxprocs?的數量等于核數，如果你的 CPU 有四個核，那么就是最多有4個?M。這樣每個核對應一個真線程，不會切換上下文，節省了一些開銷，當然你可以改變??runtime.GOMAXPROCS?的值。

type m struct { g0 *g // goroutine with scheduling stack 帶有調度堆棧信息的G morebuf gobuf // gobuf arg to morestack divmod uint32 // div/mod denominator for arm - known to liblink // Fields not known to debuggers. procid uint64 // for debuggers, but offset not hard-coded gsignal *g // signal-handling g goSigStack gsignalStack // Go-allocated signal handling stack sigmask sigset // storage for saved signal mask tls [6]uintptr // thread-local storage (for x86 extern register) mstartfn func() curg *g // current running goroutine caughtsig guintptr // goroutine running during fatal signal p puintptr // attached p for executing go code (nil if not executing go code) 這就是綁定到M的調度器P nextp puintptr oldp puintptr // the p that was attached before executing a syscall id int64 mallocing int32 throwing int32 preemptoff string // if != "", keep curg running on this m locks int32 dying int32 profilehz int32 spinning bool // m is out of work and is actively looking for work blocked bool // m is blocked on a note newSigstack bool // minit on C thread called sigaltstack printlock int8 incgo bool // m is executing a cgo call freeWait uint32 // if == 0, safe to free g0 and delete m (atomic) fastrand [2]uint32 needextram bool traceback uint8 ncgocall uint64 // number of cgo calls in total ncgo int32 // number of cgo calls currently in progress cgoCallersUse uint32 // if non-zero, cgoCallers in use temporarily cgoCallers *cgoCallers // cgo traceback if crashing in cgo call park note alllink *m // on allm schedlink muintptr mcache *mcache lockedg guintptr createstack [32]uintptr // stack that created this thread. lockedExt uint32 // tracking for external LockOSThread lockedInt uint32 // tracking for internal lockOSThread nextwaitm muintptr // next m waiting for lock waitunlockf func(*g, unsafe.Pointer) bool waitlock unsafe.Pointer waittraceev byte waittraceskip int startingtrace bool syscalltick uint32 thread uintptr // thread handle freelink *m // on sched.freem // these are here because they are too large to be on the stack // of low-level NOSPLIT functions. libcall libcall libcallpc uintptr // for cpu profiler libcallsp uintptr libcallg guintptr syscall libcall // stores syscall parameters on windows vdsoSP uintptr // SP for traceback while in VDSO call (0 if not in call) vdsoPC uintptr // PC for traceback while in VDSO call dlogPerM mOS}

其中：

curg *g // current running goroutine

表示正在真線程?M?上運行的?G。

2.4

P：將 G?調度到?M?的調度器

每一個真線程?M?都會被綁定一個調度器?P。

type p struct { id int32 status uint32 // one of pidle/prunning/... 真線程的狀態 link puintptr schedtick uint32 // incremented on every scheduler call syscalltick uint32 // incremented on every system call sysmontick sysmontick // last tick observed by sysmon m muintptr // back-link to associated m (nil if idle) mcache *mcache raceprocctx uintptr deferpool [5][]*_defer // pool of available defer structs of different sizes (see panic.go) deferpoolbuf [5][32]*_defer // Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen. goidcache uint64 goidcacheend uint64 // Queue of runnable goroutines. Accessed without lock. runqhead uint32 runqtail uint32 runq [256]guintptr // runnext, if non-nil, is a runnable G that was ready'd by // the current G and should be run next instead of what's in // runq if there's time remaining in the running G's time // slice. It will inherit the time left in the current time // slice. If a set of goroutines is locked in a // communicate-and-wait pattern, this schedules that set as a // unit and eliminates the (potentially large) scheduling // latency that otherwise arises from adding the ready'd // goroutines to the end of the run queue. runnext guintptr // Available G's (status == Gdead) gFree struct { gList n int32 } sudogcache []*sudog sudogbuf [128]*sudog tracebuf traceBufPtr // traceSweep indicates the sweep events should be traced. // This is used to defer the sweep start event until a span // has actually been swept. traceSweep bool // traceSwept and traceReclaimed track the number of bytes // swept and reclaimed by sweeping in the current sweep loop. traceSwept, traceReclaimed uintptr palloc persistentAlloc // per-P to avoid mutex _ uint32 // Alignment for atomic fields below // Per-P GC state gcAssistTime int64 // Nanoseconds in assistAlloc gcFractionalMarkTime int64 // Nanoseconds in fractional mark worker (atomic) gcBgMarkWorker guintptr // (atomic) gcMarkWorkerMode gcMarkWorkerMode // gcMarkWorkerStartTime is the nanotime() at which this mark // worker started. gcMarkWorkerStartTime int64 // gcw is this P's GC work buffer cache. The work buffer is // filled by write barriers, drained by mutator assists, and // disposed on certain GC state transitions. gcw gcWork // wbBuf is this P's GC write barrier buffer. // // TODO: Consider caching this in the running G. wbBuf wbBuf runSafePointFn uint32 // if 1, run sched.safePointFn at next safe point pad cpu.CacheLinePad}

其中?status?表示調度器的狀態：

const ( // P status // _Pidle means a P is not being used to run user code or the // scheduler. Typically, it's on the idle P list and available // to the scheduler, but it may just be transitioning between // other states. // // The P is owned by the idle list or by whatever is // transitioning its state. Its run queue is empty. // 處理器沒有運行用戶代碼或者調度器. _Pidle = iota // _Prunning means a P is owned by an M and is being used to // run user code or the scheduler. Only the M that owns this P // is allowed to change the P's status from _Prunning. The M // may transition the P to _Pidle (if it has no more work to // do), _Psyscall (when entering a syscall), or _Pgcstop (to // halt for the GC). The M may also hand ownership of the P // off directly to another M (e.g., to schedule a locked G). // 被線程 M 持有，并且正在執行用戶代碼或者調度器 _Prunning // _Psyscall means a P is not running user code. It has // affinity to an M in a syscall but is not owned by it and // may be stolen by another M. This is similar to _Pidle but // uses lightweight transitions and maintains M affinity. // // Leaving _Psyscall must be done with a CAS, either to steal // or retake the P. Note that there's an ABA hazard: even if // an M successfully CASes its original P back to _Prunning // after a syscall, it must understand the P may have been // used by another M in the interim. // 沒有執行用戶代碼，當前線程陷入系統調用 _Psyscall // _Pgcstop means a P is halted for STW and owned by the M // that stopped the world. The M that stopped the world // continues to use its P, even in _Pgcstop. Transitioning // from _Prunning to _Pgcstop causes an M to release its P and // park. // // The P retains its run queue and startTheWorld will restart // the scheduler on Ps with non-empty run queues. // 被線程 M 持有，當前處理器由于STW(垃圾回收)被停止 _Pgcstop // _Pdead means a P is no longer used (GOMAXPROCS shrank). We // reuse Ps if GOMAXPROCS increases. A dead P is mostly // stripped of its resources, though a few things remain // (e.g., trace buffers). // 當前處理器已經不被使用 _Pdead)

處理器執行用戶代碼時會?_Prunning，當M正在系統調用時，會?_Psyscall，當垃圾回收，Stop the world(STW) 時，會?_Pgcstop。

2.5

協程創建，銷毀，調度過程

詳見：src/runtime/proc.go

// Goroutine scheduler// The scheduler's job is to distribute ready-to-run goroutines over worker threads.//// The main concepts are:// G - goroutine.// M - worker thread, or machine.// P - processor, a resource that is required to execute Go code.// M must have an associated P to execute Go code, however it can be// blocked or in a syscall w/o an associated P.//// Design doc at https://golang.org/s/go11sched.

入口：

//go:linkname main_main main.mainfunc main_main()// The main goroutine.func main() { g := getg() // Racectx of m0->g0 is used only as the parent of the main goroutine. // It must not be used for anything else. g.m.g0.racectx = 0 // Max stack size is 1 GB on 64-bit, 250 MB on 32-bit. // Using decimal instead of binary GB and MB because // they look nicer in the stack overflow failure message. if sys.PtrSize == 8 { maxstacksize = 1000000000 } else { maxstacksize = 250000000 } // Allow newproc to start new Ms. mainStarted = true if GOARCH != "wasm" { // no threads on wasm yet, so no sysmon systemstack(func() { newm(sysmon, nil) }) } // Lock the main goroutine onto this, the main OS thread, // during initialization. Most programs won't care, but a few // do require certain calls to be made by the main thread. // Those can arrange for main.main to run in the main thread // by calling runtime.LockOSThread during initialization // to preserve the lock. lockOSThread() if g.m != &m0 { throw("runtime.main not on m0") } doInit(&runtime_inittask) // must be before defer if nanotime() == 0 { throw("nanotime returning zero") } // Defer unlock so that runtime.Goexit during init does the unlock too. needUnlock := true defer func() { if needUnlock { unlockOSThread() } }() // Record when the world started. runtimeInitTime = nanotime() gcenable() main_init_done = make(chan bool) if iscgo { if _cgo_thread_start == nil { throw("_cgo_thread_start missing") } if GOOS != "windows" { if _cgo_setenv == nil { throw("_cgo_setenv missing") } if _cgo_unsetenv == nil { throw("_cgo_unsetenv missing") } } if _cgo_notify_runtime_init_done == nil { throw("_cgo_notify_runtime_init_done missing") } // Start the template thread in case we enter Go from // a C-created thread and need to create a new thread. startTemplateThread() cgocall(_cgo_notify_runtime_init_done, nil) } doInit(&main_inittask) close(main_init_done) needUnlock = false unlockOSThread() if isarchive || islibrary { // A program compiled with -buildmode=c-archive or c-shared // has a main, but it is not executed. return } fn := main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime fn() if raceenabled { racefini() } // Make racy client program work: if panicking on // another goroutine at the same time as main returns, // let the other goroutine finish printing the panic trace. // Once it does, it will exit. See issues 3934 and 20018. if atomic.Load(&runningPanicDefers) != 0 { // Running deferred functions should not take long. for c := 0; c < 1000; c++ { if atomic.Load(&runningPanicDefers) == 0 { break } Gosched() } } if atomic.Load(&panicking) != 0 { gopark(nil, nil, waitReasonPanicWait, traceEvGoStop, 1) } exit(0) for { var x *int32 *x = 0 }}

我們自己的?main?包下的?main 方法入口執行前，都會先執行運行時的?The main goroutine.，類似于注入。Golang 先準備好各種資源，然后開始執行我們的方法，然后收尾。

從這里開始分析是極好的。

設計文檔在：https://golang.org/s/go11sched

作者：花花世界

出處：https://www.cnblogs.com/nima/p/11751393.html

51Reboot 運維開發

Golang 課程

K8S 課程

Python 自動化進階課程

Python 基礎實戰課程

運維前端課程

課程試聽預約請掃碼>>>

文章好看點這里

總結

以上是生活随笔為你收集整理的golang 数组 最后一个_面试必问：Golang高阶Golang协程实现原理的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。