Class Throttle

Index

Constructors

Methods

inputTokens outputTokens requests

Constructors

constructor

new Throttle(rpm: number, tpm: number, redundancy?: number): Throttle
Parameters
- rpm: number
- tpm: number
- redundancy: number = .5
Returns Throttle

Methods

inputTokens

inputTokens(deltaTokenCount: number, ctx: InferenceContext): Promise<void>
Parameters
- deltaTokenCount: number
- ctx: InferenceContext
Returns Promise<void>

outputTokens

outputTokens(deltaTokenCount: number): void
Parameters
- deltaTokenCount: number
Returns void

requests

requests(ctx: InferenceContext): Promise<void>
Parameters
- ctx: InferenceContext
Returns Promise<void>