阿里巴巴 Sentinel 落地实践–掌门1对1微服务体系 Solar

掌门1对1精耕在线教育领域，近几年业务得到了快速发展，但同时也遭遇了“成长的烦恼”。随着微服务数量不断增加，流量进一步暴增，硬件资源有点不堪重负，那么，如何实现更好的限流熔断降级等流量防护措施，这个课题就摆在了掌门人的面前。由于  Spring Cloud  体系已经演进到第二代，第一代的  Hystrix  限流熔断降级组件已经不大适合现在的业务逻辑和规模，同时它目前被  Spring Cloud  官方置于维护模式，将不再向前发展。

如何选择一个更好的限流熔断降级组件？经过对  Alibaba Sentinel 、 Resilience4j 、 Hystrix  等开源组件做了深入的调研和比较，最终选定  Alibaba Sentinel  做微服务体系  Solar  中的限流熔断降级必选组件。

Sentinel 简介

阿里巴巴中间件部门开发的新一代以流量为切入点，从流量控制、熔断降级、系统负载保护等多个维度保护服务的稳定性的分布式系统的流量防卫兵。它承接了阿里巴巴近10年的双十一大促流量的核心场景，例如秒杀（即突发流量控制在系统容量可以承受的范围）、消息削峰填谷、集群流量控制、实时熔断下游不可用应用等。

它具有非常丰富的开源生态：

它和  Hystrix  相比，有如下差异：

摘自官网 Sentinel Roadmap

关于  Sentinel  如何使用，它的技术实现原理怎样等，官方文档或者民间博客、公众号文章等可以提供非常详尽且有价值的材料，这些不在本文的讨论范围内，就不一一赘述。笔者尝试结合掌门1对1现有的技术栈以及中间件一体化的战略，并着眼于强大的  Spring Cloud Alibaba  技术生态圈展开阐释。

Sentinel 深度集成 Apollo

Sentinel  官方在  sentinel-datasource-apollo  模块中已经对  Apollo  做了一些扩展，主要实现了  Sentinel  规则的读取和订阅逻辑。这些并不够，我们需要对  Apollo  进行更深层次的集成。

摘自官网 在生产环境中使用 Sentinel

Solar SDK 环境初始化

定制  EnvironmentPostProcessor  类，实现如下：

• Sentinel Dashboard  的项目名称从  Apollo   AppId  的维度进行展示

• 根据环境  env  值读取相应的配置文件，并访问对应环境的  Sentinel Dashboard  域名 Sentinel Dashboard  在生产环境部署若干台  ECS  实例，阿里云  SLB  做负载均衡，实现对集群的水平扩展

public class SentinelClientEnvironmentPostProcessor implements EnvironmentPostProcessor {     private final ResourceLoader resourceLoader = new DefaultResourceLoader();     private static final String DEFAULT_CLASSPATH_LOCATION = "classpath:/META-INF/app.properties";     private static final String DEFAULT_LOCATION = "/META-INF/app.properties";     private static final String DEFAULT_LOG_LOCATION = "/opt/logs/";     @Override     public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {         try {             Resource appResource = resourceLoader.getResource(DEFAULT_CLASSPATH_LOCATION);             if (!appResource.exists()) {                 appResource = resourceLoader.getResource(DEFAULT_LOCATION);             }             Properties appProperties = new Properties();             appProperties.load(new InputStreamReader(appResource.getInputStream()));             String appId = appProperties.getProperty("app.id");             System.setProperty("project.name", appId);             System.setProperty("csp.sentinel.log.dir", DEFAULT_LOG_LOCATION + appId);             Properties properties = new Properties();             String path = isOSWindows() ? "C:/opt/settings/server.properties" : "/opt/settings/server.properties";             File file = new File(path);             if (file.exists() && file.canRead()) {                 FileInputStream fis = new FileInputStream(file);                 if (fis != null) {                     try {                         properties.load(new InputStreamReader(fis, Charset.defaultCharset()));                     } finally {                         fis.close();                     }                 }             }             String idc = properties.getProperty("idc");             String location;             String env = System.getProperty("env");             if (StringUtils.isEmpty(idc)) {                 if (!isBlank(env)) {                     env = env.trim().toLowerCase();                 } else {                     env = System.getenv("ENV");                     if (!isBlank(env)) {                         env = env.trim().toLowerCase();                     } else {                         env = properties.getProperty("env");                         if (!isBlank(env)) {                             env = env.trim();                         } else {                             env = Env.FAT.getEnv();                         }                     }                 }                 location = "classpath:/META-INF/sentinel-" + env + ".properties";             } else {                 location = "classpath:/META-INF/sentinel-" + idc + ".properties";             }             Resource serverResource = resourceLoader.getResource(location);             properties.load(new InputStreamReader(serverResource.getInputStream()));             for (String key : properties.stringPropertyNames()) {                 System.setProperty(key, properties.getProperty(key));             }             System.setProperty(CommonConstant.SENTINEL_VERSION_NAME, CommonConstant.SENTINEL_VERSION_VALUE);         } catch (Exception e) {             LOG.error(e.getMessage());         }     }     private boolean isBlank(String str) {         return Strings.nullToEmpty(str).trim().isEmpty();     }     private boolean isOSWindows() {         String osName = System.getProperty("os.name");         return !isBlank(osName) && osName.startsWith("Windows");     } }

把  SentinelClientEnvironmentPostProcessor  类放置  \resources\META-INF\spring.factories  文件中，内容为

org.springframework.boot.env.EnvironmentPostProcessor=\ com.zhangmen.solar.component.sentinel.common.context.SentinelClientEnvironmentPostProcessor

在  \resources\META-INF  目录下，定制环境配置文件，文件名格式为  sentinel-{环境号}.properties 。下文以  dev  环境和  flow  流控配置（其它规则配置，请自行参考  Spring Cloud Alibaba Sentinel  的相关资料）为样例。

sentinel-dev.properties

spring.cloud.sentinel.transport.dashboard=127.0.0.1:8080 spring.cloud.sentinel.datasource.ds.apollo.namespaceName=application spring.cloud.sentinel.datasource.ds.apollo.flowRulesKey=sentinel.flowRules spring.cloud.sentinel.datasource.ds.apollo.ruleType=flow ...

Sentinel Dashboard 持久化改造

原生的  Sentinel Dashboard  在创建完规则后，规则内容保存在服务的内存中，当服务重启后所有的规则内容都会消失。因此，在生产部署时需要考虑配置持久化，并且使用  Apollo  动态规则的感知能力。

① 向外暴露 Sentinel 规则的 Restful 接口

@RestController @RequestMapping(value = "/v2/flow") public class FlowControllerV2 {     @Autowired     @Qualifier("apolloFlowRuleProvider")     private DynamicRuleProvider<List> ruleProvider;     @Autowired     @Qualifier("apolloFlowRulePublisher")     private DynamicRulePublisher<List> rulePublisher;     .... }

② 实现 Sentinel Apollo 规则提供

@Component("apolloFlowRuleProvider") public class ApolloFlowRuleProvider extends BaseApolloRuleProvider {     @Override     public List getRules(String appName) throws Exception {         List flowRuleEntityList = super.getRules(appName);         if (!CollectionUtils.isEmpty(flowRuleEntityList)) {             List flowRuleEntities = JSONArray.parseArray(flowRuleEntityList.toString(), FlowRuleEntity.class);             long id = 1;             for (FlowRuleEntity entity : flowRuleEntities) {                 entity.setId(id++);                 entity.getClusterConfig().setFlowId(entity.getId());             }             return flowRuleEntities;         } else {             return null;         }     }     @Override     protected String getDataId() {         return ApolloConfigUtil.getFlowDataId();     } }

③ 实现 Sentinel Apollo 规则订阅

@Component("apolloFlowRulePublisher") public class ApolloFlowRulePublisher extends BaseApolloRulePublisher<List> {     @Override     public void publish(String app, String operator, List rules) throws Exception {         if (!CollectionUtils.isEmpty(rules)) {             for (int i = 0; i < rules.size(); i++) {                 rules.get(i).setId((long) (i + 1));                 rules.get(i).setApp(null);                 rules.get(i).setGmtModified(null);                 rules.get(i).setGmtCreate(null);                 rules.get(i).setIp(null);                 rules.get(i).setPort(null);                 rules.get(i).getClusterConfig().setFlowId((long) (i + 1));             }         } else {             rules = null;         }         super.publish(app, operator, rules);     }     @Override     protected String getDataId() {         return ApolloConfigUtil.getFlowDataId();     } }

上述代码实现了对  Apollo  配置读写操作。熟悉  Apollo  的同学应该知道，这些操作需要基于  Apollo OpenApi  来操作；动态感知能力的逻辑已经由  sentinel-datasource-apollo  模块实现。

Sentinel 集成 Skywalking

由于掌门1对1微服务技术栈落地的比较早，鉴于历史的局限性（当时没有更先进的技术可供选择），除了 Hystrix 比较古老以外，另一个技术栈的痛点是全链路监控中间件的改造也提上议事日程，CAT 作为开源界老牌作品，为公司底层全链路监控提供强有力的保障，但随着技术的演进，它逐渐已经不适合公司的未来发展方向，经过对比，最终选择 Skywalking 将作为它的替代者（关于 Skywalking 的技术选型，将在后面掌门1对1微服务体系 Solar 的公众号系列文章中会一一阐述）。

业务系统要求对限流熔断降级实现全链路实时埋点，并希望在  Skywalking  界面上提供限流熔断降级埋点的多维度统计。由于  Skywalking  实现了  OpenTracing  标准化协议，那么以  OpenTracing  为桥梁，通过  Solar  SDK 输出  Sentinel  埋点到  Skywalking  Server 不失为一个好的技术选择。下面简单扼要介绍一下基于  Sentinel InitFunc  的  SPI  机制实现埋点输出：

Sentinel  将  ProcessorSlot  作为  SPI  接口进行扩展（1.7.2 版本以前  SlotChainBuilder  作为  SPI ），使得  Slot Chain  具备了扩展的能力。您可以自行加入自定义的 slot 并编排 slot 间的顺序，从而可以给  Sentinel  添加自定义的功能。

摘自官网 Sentinel 工作主流程

抽象 Sentinel ProcessorSlot 埋点输出

Sentinel  的  ProcessorSlotEntryCallback  提供  onPass  和  onBlocked  两个方法，毕竟限流熔断降级并不是常规的功能，不会发生在大流量上面，所以  onPass  上我们不做任何处理，否则正常的调用去实现拦截，将为产生大量的埋点数据，会让  Skywalking  Server 承受很大的性能压力，所以  onBlocked  将是我们关注的重点，它除了输出  Sentinel  本身的上下文参数之外，也会输出微服务  Solar   指标参数，主要包括：

• 埋点  Span 名称，这里为  SENTINEL  ，在  Skywalking  全链路监控界面中，用户可以非常容易的找到这个埋点

• 服务所在的 组 名，指服务的 逻辑分组

• 服务类型，包括服务和网关（网关也是一种特殊的服务），  Sentinel  埋点可以支持在服务和网关上的输出

• 服务的  APPID ，它为  Apollo  组件的范畴概念

• 服务名，它对应为  spring.application.name  的配置值

• 服务实例所在的  IP  地址和  Port  端口

• 服务版本号

• 服务所在的区域

• 服务所在的子环境

接下去是  Sentinel  层面的参数，请自行参考  Sentinel  官方文档和源码，了解其含义，这里不做具体讲解。

public abstract class SentinelTracerProcessorSlotEntryCallback implements ProcessorSlotEntryCallback {     @Override     public void onPass(Context context, ResourceWrapper resourceWrapper, DefaultNode param, int count, Object... args) throws Exception {     }     @Override     public void onBlocked(BlockException e, Context context, ResourceWrapper resourceWrapper, DefaultNode param, int count, Object... args) {         S span = buildSpan();         PluginAdapter pluginAdapter = PluginContextAware.getStaticApplicationContext().getBean(PluginAdapter.class);         outputSpan(span, DiscoveryConstant.SPAN_TAG_PLUGIN_NAME, context.getName());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_GROUP, pluginAdapter.getGroup());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_TYPE, pluginAdapter.getServiceType());         String serviceAppId = pluginAdapter.getServiceAppId();         if (StringUtils.isNotEmpty(serviceAppId)) {             outputSpan(span, DiscoveryConstant.N_D_SERVICE_APP_ID, serviceAppId);         }         outputSpan(span, DiscoveryConstant.N_D_SERVICE_ID, pluginAdapter.getServiceId());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_ADDRESS, pluginAdapter.getHost() + ":" + pluginAdapter.getPort());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_VERSION, pluginAdapter.getVersion());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_REGION, pluginAdapter.getRegion());         outputSpan(span, DiscoveryConstant.N_D_SERVICE_ENVIRONMENT, pluginAdapter.getEnvironment());         outputSpan(span, SentinelStrategyConstant.ORIGIN, context.getOrigin());         outputSpan(span, SentinelStrategyConstant.ASYNC, String.valueOf(context.isAsync()));         outputSpan(span, SentinelStrategyConstant.RESOURCE_NAME, resourceWrapper.getName());         outputSpan(span, SentinelStrategyConstant.RESOURCE_SHOW_NAME, resourceWrapper.getShowName());         outputSpan(span, SentinelStrategyConstant.RESOURCE_TYPE, String.valueOf(resourceWrapper.getResourceType()));         outputSpan(span, SentinelStrategyConstant.ENTRY_TYPE, resourceWrapper.getEntryType().toString());         outputSpan(span, SentinelStrategyConstant.RULE_LIMIT_APP, e.getRuleLimitApp());         if (tracerSentinelRuleOutputEnabled) {             outputSpan(span, SentinelStrategyConstant.RULE, e.getRule().toString());         }         outputSpan(span, SentinelStrategyConstant.CAUSE, e.getClass().getName());         outputSpan(span, SentinelStrategyConstant.BLOCK_EXCEPTION, e.getMessage());         outputSpan(span, SentinelStrategyConstant.COUNT, String.valueOf(count));         if (tracerSentinelArgsOutputEnabled) {             outputSpan(span, SentinelStrategyConstant.ARGS, JSON.toJSONString(args));         }         finishSpan(span);     }     protected abstract S buildSpan();     protected abstract void outputSpan(S span, String key, String value);     protected abstract void finishSpan(S span); }

public class SentinelSkywalkingTracerProcessorSlotEntryCallback extends SentinelTracerProcessorSlotEntryCallback {     private Tracer tracer = new SkywalkingTracer();         @Override     protected Span buildSpan() {         return tracer.buildSpan(SentinelStrategyConstant.SPAN_NAME).startManual();     }     @Override     protected void outputSpan(Span span, String key, String value) {         span.setTag(key, value);     }     @Override     protected void finishSpan(Span span) {         span.finish();     } }

public class SentinelSkywalkingTracerInitFunc implements InitFunc {     @Override     public void init() throws Exception {         StatisticSlotCallbackRegistry.addEntryCallback(SentinelSkywalkingTracerProcessorSlotEntryCallback.class.getName(), new SentinelSkywalkingTracerProcessorSlotEntryCallback());     } }

com.nepxion.discovery.plugin.strategy.sentinel.skywalking.monitor.SentinelSkywalkingTracerInitFunc

@Component public class MetricFetcher {     @Autowired     @Qualifier("influxDBMetricRepository")     private MetricsRepository metricStore;     ... }

@Configuration public class InfluxDBAutoConfiguration {     @Value("${spring.influx.url}")     private String influxDBUrl;     @Value("${spring.influx.user}")     private String userName;     @Value("${spring.influx.password}")     private String password;     @Value("${spring.influx.database}")     private String database;     @Bean     public InfluxDB influxDB() {         InfluxDB influxDB = null;         try {             influxDB = InfluxDBFactory.connect(influxDBUrl, userName, password);             influxDB.setDatabase(database).enableBatch(100, 1000, TimeUnit.MILLISECONDS);             influxDB.setLogLevel(InfluxDB.LogLevel.NONE);         } catch (Exception e) {             LOG.error(e.getMessage());         }         return influxDB;     } }

@Component("influxDBMetricRepository") public class InfluxDBMetricRepository implements MetricsRepository {     @Autowired     private InfluxDB influxDB;     @Override     public void save(MetricEntity metric) {         try {             Point point = createPoint(metric);             influxDB.write(point);         } catch (Exception e) {             LOG.error(e.getMessage());         }     }     @Override     public void saveAll(Iterable metrics) {         if (metrics == null) {             return;         }         try {             BatchPoints batchPoints = BatchPoints.builder().build();             metrics.forEach(metric -> {                 Point point = createPoint(metric);                 batchPoints.point(point);             });             influxDB.write(batchPoints);         } catch (Exception e) {             LOG.error(e.getMessage());         }     } }

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-id

[     {         "resource": "sentinel-resource",         "limitApp": "a-service-id",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-app-id

[     {         "resource": "sentinel-resource",         "limitApp": "a-service-app-id",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-address

[     {         "resource": "sentinel-resource",         "limitApp": "a-ip:a-port",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-group

[     {         "resource": "sentinel-resource",         "limitApp": "b-group",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-version

    {         "resource": "sentinel-resource",         "limitApp": "b-version",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-region

[     {         "resource": "sentinel-resource",         "limitApp": "b-region",         "strategy": 0     } ]

spring.application.strategy.service.sentinel.request.origin.key=n-d-service-env

[     {         "resource": "sentinel-resource",         "limitApp": "b-env",         "strategy": 0     } ]

public class MyServiceSentinelRequestOriginAdapter extends AbstractServiceSentinelRequestOriginAdapter {     @Override     public String parseOrigin(HttpServletRequest request) {         return request.getHeader("userName");     } }

@Bean public ServiceSentinelRequestOriginAdapter ServiceSentinelRequestOriginAdapter() {     return new MyServiceSentinelRequestOriginAdapter(); }

[     {         "resource": "sentinel-resource",         "limitApp": "zhangsan",         "strategy": 0     } ]

public class MyServiceSentinelRequestOriginAdapter extends AbstractServiceSentinelRequestOriginAdapter {     @Override     public String parseOrigin(HttpServletRequest request) {         String userName = request.getHeader("userName");         String userType = getUserTypeByName(userName);         return userType;     } }

[     {         "resource": "sentinel-resource",         "limitApp": "AUTH_USER",         "strategy": 0     } ]

public class MyServiceSentinelRequestOriginAdapter extends AbstractServiceSentinelRequestOriginAdapter {     @Override     public String parseOrigin(HttpServletRequest request) {         String version = request.getHeader(DiscoveryConstant.N_D_SERVICE_VERSION);         String userName = request.getHeader("userName");         return version + "&" + userName;     } }

[     {         "resource": "sentinel-resource",         "limitApp": "1.0&zhangsan",         "strategy": 0     } ]

public class SentinelApolloTokenClusterInitFunc implements InitFunc {     @Override     public void init() throws Exception {         // Register client dynamic rule data source.         initDynamicFlowRuleProperty();         initDynamicParamRuleProperty();         // Register token client related data source.         // Token client common config:         ClusterClientConfigInitializer.doInit();         // Token client assign config (e.g. target token server) retrieved from assign map:         ClusterClientAssignConfigInitializer.doInit();         // Register token server related data source.         // Register dynamic rule data source supplier for token server:         ClusterRuleSupplierInitializer.doInit();         // Token server transport config extracted from assign map:         ServerTransportConfigInitializer.doInit();         // Init cluster state property for extracting mode from cluster map data source.         ClusterStateInitializer.doInit();         // ServerFlowConfig 配置         ServerFlowConfigInitializer.doInit();     } }

com.zhangmen.solar.sentinel.SentinelApolloTokenClusterInitFunc